Compare commits
29 Commits
ee67b6370a
...
bc03adb1fd
| Author | SHA1 | Date | |
|---|---|---|---|
| bc03adb1fd | |||
| 099bc1b6e0 | |||
| 57deaeba16 | |||
| 7c80d36eb6 | |||
| 062086f9a6 | |||
| cfd403710e | |||
| 699871040a | |||
| 87e8aff786 | |||
| 984e069f5d | |||
| e583b068a1 | |||
| 99c13e4459 | |||
| 3b739b982b | |||
| 56bb4855bb | |||
| 1f88669304 | |||
| db6446ce73 | |||
| b028322568 | |||
| d803e0eab4 | |||
| 50b504d293 | |||
| acce99de9a | |||
| 91434e7f61 | |||
| b3db64af6c | |||
| e4123ada91 | |||
| 2a8f9af5e4 | |||
| 1b2c36eb87 | |||
| 924eff8d3c | |||
| f9fd495292 | |||
| 7d75478f10 | |||
| e5d12d391a | |||
| db6a53036b |
@@ -302,7 +302,9 @@ non-`Healthy` state **and** its `LastSuccessfulRead` has been frozen for 90 s. T
|
||||
success. Each conjunct has a test that goes red when the *other* is deleted.
|
||||
- **The mechanism rests on every driver reaching a non-Healthy state on sustained failure.** Surveyed across
|
||||
all 12: nine poll drivers via `Degraded`, MQTT via live connection state, Galaxy via its reconnect
|
||||
supervisor. `CalculationDriver` reports `Healthy` while running and so can never trip — correct, as it has
|
||||
supervisor, OpcUaClient via a bad session keep-alive (#670 — it used to stay `Healthy` forever; its SDK
|
||||
`SessionReconnectHandler` is deleted, a lost session publishes `BadNoCommunication` once per outage and the
|
||||
watchdog's re-init is the recovery; good keep-alives + notifications advance `LastSuccessfulRead`). `CalculationDriver` reports `Healthy` while running and so can never trip — correct, as it has
|
||||
no transport. Re-check this if a new driver lands.
|
||||
- **The reconnect retry MUST tear down before re-initialising — FIXED in both drivers, keep it that way.**
|
||||
`ModbusDriver` (`_transport`) and `MqttDriver` (`_connection`) used to overwrite an `IAsyncDisposable`
|
||||
@@ -1341,11 +1343,12 @@ historian tagname, which is the tag's driver `FullName`.
|
||||
### Pluggable backends, SQL Server and HistoryUpdate (2026-10)
|
||||
|
||||
- **Selection:** `ServerHistorian:Backend` (`Gateway` default | `SqlServer` | `QuestDb`; unknown fails host start). `HistorianBackendRegistry` → `IHistorianBackendProvider` builds all five seams; the gateway keys are only required for `Gateway`. `SqlServer` uses `ServerHistorian:SqlServer:{ConnectionString (required), Schema=hist, CommandTimeout=30s, AutoCreateSchema=true}`.
|
||||
- **SQL Server** (`Driver.Historian.SqlServer`): `hist.Sample` PK `(TagId, SourceTimeUtc)` — writes are insert-if-absent, so outbox redelivery is idempotent. Stores every value kind with its **real status**; aggregates (13, simple per-sample definitions, not Part 13) are pushed down to T-SQL with bucket math in **100 ns ticks**. Docker-dev recipe, schema and aggregate semantics: `docs/Historian.md` §"SQL Server backend". Integration tests need `HISTORIAN_SQL_TEST_CONNECTION`.
|
||||
- **SQL Server** (`Driver.Historian.SqlServer`): `hist.Sample` PK `(TagId, SourceTimeUtc)` — writes are insert-if-absent, so outbox redelivery is idempotent. Stores every value kind with its **real status**; per-sample aggregates (12, simple definitions) are pushed down to T-SQL with bucket math in **100 ns ticks**; the Part 13 time-weighted five (TimeAverage/TimeAverage2/Total/Total2/Interpolative, #671 — ⚠️ `Total` is now the time-weighted integral, no longer `SUM`) stream window + 4 bound rows into the shared `TimeWeightedAggregator` (both SQL backends, identical results; client `AggregateConfiguration` ignored; status = the SDK `AggregateCalculator` time-based rule, Uncertain time counts as Good; reverse `Interpolative` anchored at the later time; window capped by `ServerHistorian:MaxTimeWeightedRawRows` ⇒ `BadTooManyOperations`). Docker-dev recipe, schema and aggregate semantics: `docs/Historian.md` §"SQL Server backend". Integration tests need `HISTORIAN_SQL_TEST_CONNECTION`. **Retention (#667):** opt-in `SqlServer:RetentionDays` (+ `PurgeInterval` 1h, `PurgeBatchSize` 4000) -> `HistorianRetentionService` purges old `hist.Sample` (per tag) / `hist.Event` in `DELETE TOP(n)` batches, Primary-gated on `IHistorizationRoleView`, one purger per DB via `sp_getapplock`; counter `otopcua.historian.retention.purged{table}`. Optional `IHistorianBackendProvider.CreateRetention` (default null). `docs/Historian.md` §Retention.
|
||||
- **QuestDB** (`Driver.Historian.QuestDb`, `ServerHistorian:QuestDb:*`, Npgsql over PG wire; docker-dev overlay `docker-compose.questdb.yml`): same kinds/status/aggregates/HistoryUpdate as SqlServer, soft delete by tombstone (INSERT only — never `UPDATE`/`DELETE`; every read filters `NOT deleted`). ⚠️ **Timestamps are µs** (`QuestDbConnection.Round`/`BindTimestamp`; `IHistorianDataSource.TimestampResolutionTicks` = 10 steps HistoryRead paging), **pre-1970 cannot be stored** (writer drops, HistoryUpdate `BadOutOfRange`), every write waits for WAL visibility, TTL is enforced on write commits only, set Npgsql `Maximum Pool Size` < QuestDB's 64. `docs/Historian.md` §"QuestDB backend".
|
||||
- **Real status fix:** the recorder used to stamp every sample Good (`192`); it now carries the driver's status. ⚠️ On the **Gateway** backend this changes nothing stored: the gateway discards quality (C-002), so AVEVA still stores Good.
|
||||
- **Comms loss is historized as an outage marker** (`Empty` + `BadNoCommunication`, one per tag per outage) on SqlServer; dropped (`DroppedUnsupportedKind`) on Gateway. Outbox records are **format v2** — drain or delete `ContinuousHistorization:OutboxPath` before downgrading a node.
|
||||
- **HistoryUpdate** (SqlServer only; Gateway ⇒ `BadHistoryOperationUnsupported`): Insert/Replace/Update/DeleteRaw/DeleteAtTime, gated in both ACL modes — flag off needs the `HistoryUpdate` role from a **Cluster-scoped** grant (+ key re-check), flag on uses `DataPlaneAuthorizer`'s HistoryUpdate operation; `WriteOperate` never implies it. The `HistoryWrite` AccessLevel bit is set only when an updater is wired at materialise time. `docs/Historian.md` §HistoryUpdate, `docs/security.md`.
|
||||
- **Modified-value history (#668, Part 11):** every applied HistoryUpdate item leaves a trail row (prior value, principal from the session's API key / `anonymous`, `HistoryModificationType`, time) — SqlServer `hist.SampleModified` (**schema v2**, written by `OUTPUT … INTO` in the same statement ⇒ atomic; ⚠️ no FK/CHECK/trigger on it, and `AutoCreateSchema=false` installs must run `002_sample_modified.sql` before upgrading), QuestDB `{p}sample_modified` (append-only, written **before** the change ⇒ a crash leaves an orphan trail row, never an untracked change). `IsReadModified=true` HistoryRead serves it as `HistoryModifiedData` via `IHistorianDataSource.ReadModifiedAsync` (default ⇒ `NotSupported` ⇒ `BadHistoryOperationUnsupported` on Gateway/Null), paged on `(SourceTime, ModifiedId)`, gated by plain HistoryRead. Live writes and retention never write it; retention/TTL purge it on source time. `IHistorianDataUpdater` methods take a `HistoryUpdateContext`. CLI `historyread --modified`. `docs/Historian.md` §Modified-value history.
|
||||
|
||||
### Read path (`ServerHistorian` section)
|
||||
|
||||
@@ -1370,6 +1373,7 @@ authorization uses the standard `AccessLevels.HistoryRead` bit set at materializ
|
||||
| `MaxTieClusterOverfetch` | `65536` | Bounded over-fetch the HistoryRead-Raw paging uses to page within an oversized same-timestamp tie cluster (retained from the prior backend) |
|
||||
| `HistoryReadBatchParallelism` | `4` | Max nodes served concurrently within one HistoryRead batch (03/S3); `⌈N/P⌉ × CallTimeout` worst case instead of sequential |
|
||||
| `MaxConcurrentHistoryReadBatches` | `16` | Process-wide concurrent HistoryRead batch cap (03/S3); flood degrades to `BadTooManyOperations`. Total in-flight reads capped at 16 × 4 = 64 |
|
||||
| `MaxTimeWeightedRawRows` | `1000000` | `SqlServer`/`QuestDb` only (#671): raw in-window rows one time-weighted aggregate read (incl. `Total`) may stream into the C# evaluator; window query capped at this + 1, past it ⇒ `BadTooManyOperations` for that node. ≤ 0 = unbounded |
|
||||
| `HistoryReadDeadline` | `00:01:00` | Per-request HistoryRead wall-clock deadline (03/S3); a node still in flight surfaces `BadTimeout`. Non-positive = unbounded |
|
||||
|
||||
### Alarm-history path (`AlarmHistorian` section)
|
||||
@@ -1408,6 +1412,7 @@ only the recorder + outbox knobs:
|
||||
| `MaxWriteAttempts` | `5` | #538: consecutive failed drain passes one tag may accumulate — counted **only across passes where other tags wrote successfully** — before its queued entries are abandoned so the FIFO head can advance |
|
||||
| `MinBackoffSeconds` | `1` | Initial retry backoff after a failed drain pass |
|
||||
| `MaxBackoffSeconds` | `30` | Cap on the exponential retry backoff |
|
||||
| `CaptureWhilePrimaryUnreachable` | `true` | #672: a standby stood down for its mesh Primary (#537) captures as soon as that Primary is reported UNREACHABLE, instead of after auto-down — crash-failover gap ~30 s ⇒ ~10–12 s, at the cost of duplicate windows (see #537/#672 below). `false` = pre-#672 |
|
||||
|
||||
### Tag auto-provisioning (`IHistorianProvisioning` EnsureTags hook)
|
||||
|
||||
@@ -1547,6 +1552,21 @@ tag); evidence in the same plan doc. Two things to know before touching either.*
|
||||
`RedundancyRoleView` behind **`IHistorizationRoleView`**, registered unconditionally on driver nodes
|
||||
(NOT alongside the alarm `IRedundancyRoleView`, which only exists when that section is enabled — the
|
||||
#511 registration accident).
|
||||
- **#672 — an UNREACHABLE recording Primary does not count (crash-failover gap).** A SIGKILLed Primary
|
||||
stays a member (unreachable, not downed) for 10 s failure detection + 15 s auto-down, and the
|
||||
redundancy singleton died with it, so the standby's cached snapshot kept naming it Primary and
|
||||
*nobody* captured for ~30 s. `DriverHostActor` now holds its **own** `ReachabilityEvent` (+`MemberUp`)
|
||||
subscription — not inside `StartPrimaryVerdictPublishing`, which early-returns with no primary-role
|
||||
source — handled in **all three** behaviours, caches the last snapshot, and on a reachability change
|
||||
re-derives and republishes **only** the historization verdict with unreachable peers excluded. Gap ⇒ the
|
||||
failure-detector window (~10–12 s). Gated by `ContinuousHistorization:CaptureWhilePrimaryUnreachable`
|
||||
(default `true`). ⚠️ **The alarm drain and `ShouldServiceAsPrimary`/`PublishPrimaryVerdict` must NOT
|
||||
adopt this** (a false allow there is a dual claimant, not a duplicate) — the tests assert both stay
|
||||
put. ⚠️ A **removed** member deliberately stays marked unreachable (only `MemberUp` at that address
|
||||
clears it): un-marking it would re-derive from a cached snapshot still naming it Primary and close the
|
||||
gate on a dead node. Cost: duplicates while a live Primary is merely unreachable and under a real
|
||||
partition — idempotent on SqlServer/QuestDB only for source-timestamped drivers (Modbus stamps each
|
||||
node's own poll time ⇒ duplicate rows on every backend); Gateway stores duplicates.
|
||||
|
||||
Re-confirmed live in 2026-06 and **since fixed (#600.1)**: `EnsureTags` provisioning used to fire only for
|
||||
newly-ADDED historized tags, so flipping an existing tag's `historianTagname` (or historization, or DataType)
|
||||
|
||||
@@ -210,8 +210,19 @@ otopcua-cli historyread -u opc.tcp://localhost:4840/OtOpcUa \
|
||||
-n "ns=1;s=TestMachine_001.TestHistoryValue" \
|
||||
--start "2026-03-25T00:00:00Z" --end "2026-03-30T00:00:00Z" \
|
||||
--aggregate Average --interval 3600000
|
||||
|
||||
# Modified-value history (what HistoryUpdate changed): prior value, update type, user, modification time
|
||||
otopcua-cli historyread -u opc.tcp://localhost:4840/OtOpcUa -U apikey -P otua_<keyId>_<secret> \
|
||||
-n "ns=2;s=Plant/Modbus/dev1/Speed" \
|
||||
--start "2026-01-15T00:00:00Z" --end "2026-01-16T00:00:00Z" --modified
|
||||
```
|
||||
|
||||
`--modified` sends `ReadRawModifiedDetails` with `IsReadModified = true` (OPC UA Part 11, Gitea #668) and prints one line per
|
||||
recorded modification: the source timestamp, the **prior** value (`(none)` for an insert, which had none), its status, the update type
|
||||
(`Insert` / `Replace` / `Update` / `Delete`), the user (the API key's principal, or `anonymous`) and the modification time. It follows
|
||||
continuation points up to `--max`. A server on the `Gateway` backend (no modification trail) answers `BadHistoryOperationUnsupported`.
|
||||
`--modified` cannot be combined with `--aggregate`. See [Historian.md §Modified-value history](Historian.md#modified-value-history-part-11).
|
||||
|
||||
| Flag | Description |
|
||||
|------|-------------|
|
||||
| `-n` / `--node` | Node ID to read history for (required) |
|
||||
@@ -220,6 +231,7 @@ otopcua-cli historyread -u opc.tcp://localhost:4840/OtOpcUa \
|
||||
| `--max` | Maximum number of values (default: 1000) |
|
||||
| `--aggregate` | Aggregate function: Average, Minimum, Maximum, Count, Start, End, StandardDeviation |
|
||||
| `--interval` | Processing interval in milliseconds for aggregates (default: 3600000) |
|
||||
| `--modified` | Read modified-value history (Part 11 `IsReadModified`) instead of raw samples |
|
||||
|
||||
### historyupdate
|
||||
|
||||
|
||||
@@ -411,6 +411,9 @@ Validated at host start (driver-role nodes) when `ServerHistorian:Enabled` **or*
|
||||
| `ServerHistorian:SqlServer:Schema` | `hist` | Must match `^[A-Za-z_][A-Za-z0-9_]{0,127}$` (spliced into DDL/DML). |
|
||||
| `ServerHistorian:SqlServer:CommandTimeout` | `00:00:30` | Must be `> 0`; rounded up to whole seconds. |
|
||||
| `ServerHistorian:SqlServer:AutoCreateSchema` | `true` | `true` applies the embedded schema script under an app-lock on first use; `false` only verifies `SchemaVersion >= 1` and fails every historian call (health `LastError`, no crash) if behind. |
|
||||
| `ServerHistorian:SqlServer:RetentionDays` | unset | Days of history kept (source time), `>= 1` when set; unset keeps everything. Enables the Primary-gated purge of `hist.Sample`/`hist.Event` (see [Historian.md](Historian.md#retention-retentiondays)). |
|
||||
| `ServerHistorian:SqlServer:PurgeInterval` | `01:00:00` | Purge cadence; must be `> 0`. |
|
||||
| `ServerHistorian:SqlServer:PurgeBatchSize` | `4000` | Rows per `DELETE TOP (n)` batch; `>= 1`. Keep under ~5000 to avoid lock escalation. |
|
||||
|
||||
#### `ServerHistorian:QuestDb` (`Backend=QuestDb`)
|
||||
|
||||
@@ -447,6 +450,7 @@ Only the knobs; the backend connection comes from `ServerHistorian`. Outbox reco
|
||||
| `ContinuousHistorization:DrainConcurrency` | `8` | Historian tags one drain pass writes **concurrently** — a batch is grouped per tag and each group is its own gateway round-trip, so this turns a sequential `tags × latency` pass into `⌈tags / concurrency⌉ × latency` while bounding outstanding round-trips. `1` = fully sequential; `< 1` is clamped to 1 with a `Validate()` warning. The ack is unchanged (all-or-nothing after every group reports), as is the #538 quarantine rule. |
|
||||
| `ContinuousHistorization:Capacity` | `500000` | Max un-acked entries before drop-oldest; `0` = unbounded. The #538 disk backstop for a historian refusing **every** write — the one case the per-tag quarantine cannot resolve. |
|
||||
| `ContinuousHistorization:MaxWriteAttempts` | `5` | #538 poison-tag budget: consecutive failed passes one tag may accumulate — **counted only across passes where *other* tags wrote successfully** — before its entries are abandoned so the FIFO head can advance. |
|
||||
| `ContinuousHistorization:CaptureWhilePrimaryUnreachable` | `true` | #672: a standby stood down for its mesh-partner Primary (#537) resumes capture as soon as the failure detector reports that Primary **unreachable**, rather than after auto-down + a fresh snapshot — a crash-failover gap of ~25–30 s becomes the failure-detector window (~10–12 s). Cost: duplicate capture while a live Primary is merely unreachable and under a real partition (absorbed idempotently on SqlServer/QuestDB only for source-timestamped drivers; Modbus-style poll-time stamps and the Gateway backend store duplicates). Affects only the capture verdict. `false` = pre-#672 behaviour. |
|
||||
| `ContinuousHistorization:MinBackoffSeconds` / `MaxBackoffSeconds` | `1` / `30` | Initial and capped exponential retry backoff after a failed drain pass. |
|
||||
|
||||
### `LocalDb` (pair-local config cache — driver role only)
|
||||
|
||||
@@ -422,7 +422,10 @@ Two consequences worth knowing:
|
||||
- **`CalculationDriver` can never trip it, correctly.** It reports `Healthy` whenever it is running and has
|
||||
no transport to be degraded about. Every other driver reaches a non-Healthy state on sustained failure:
|
||||
the nine poll drivers via `Degraded`, MQTT via its live connection state, Galaxy via its reconnect
|
||||
supervisor. This was surveyed across all 12 before relying on it — it is the assumption the whole
|
||||
supervisor, OpcUaClient via a bad session keep-alive (`Degraded`, since Gitea #670 — before that its health
|
||||
only moved on `ReadAsync`, which a subscription-only driver never calls, so it stayed `Healthy` through any
|
||||
outage and this survey was wrong about it; good keep-alives and notifications now advance its read stamp, so
|
||||
an idle-but-alive subscription does not look dead either). This was surveyed across all 12 before relying on it — it is the assumption the whole
|
||||
mechanism rests on.
|
||||
- **A device that accepts a connection but serves no data will cycle**, reconnecting once per threshold.
|
||||
That is the correct reading of a broken device, it is rate-limited, and it is visible in the health
|
||||
|
||||
+277
-37
@@ -73,12 +73,13 @@ consumes the historian. The existing enable gates (`ServerHistorian:Enabled`, `A
|
||||
|
||||
| | `Gateway` (default) | `SqlServer` | `QuestDb` |
|
||||
|---|---|---|---|
|
||||
| Storage | AVEVA Historian via the HistorianGateway sidecar | `{Schema}` tables in a SQL Server database you point it at | `{prefix}sample` / `event` / `tag` WAL tables in a QuestDB instance |
|
||||
| Storage | AVEVA Historian via the HistorianGateway sidecar | `{Schema}` tables in a SQL Server database you point it at | `{prefix}sample` / `event` / `tag` / `sample_modified` WAL tables in a QuestDB instance |
|
||||
| Value kinds recorded | `Double`, `Int64`, `Boolean` (numeric analog only) | all: `Double`, `Int64`, `Boolean`, `String`, `DateTime`, plus `Empty` (a Bad sample with no value) | same as `SqlServer` |
|
||||
| Status code stored | Sent as DA quality but **discarded** by the gateway (C-002); AVEVA stores every sample Good | Stored bit-for-bit (`StatusCode` int column) | Stored bit-for-bit (unsigned, in a `LONG`) |
|
||||
| Comms-loss / driver-Bad publishes | Dropped, counted in `DroppedUnsupportedKind` | Stored as an `Empty` outage-marker sample carrying the Bad code (see [below](#typed-values-real-status-and-outage-markers)) | Stored as an `Empty` outage-marker sample, as on `SqlServer` |
|
||||
| Processed aggregates | Average, Minimum, Maximum, Total, Count | The thirteen in the [aggregate table](#aggregates-processed-reads) | The thirteen, pushed down to QuestDB SQL |
|
||||
| Processed aggregates | Average, Minimum, Maximum, Total, Count | The seventeen in the [aggregate table](#aggregates-processed-reads) (incl. Part 13 time-weighted, #671) | The same seventeen, identical results |
|
||||
| HistoryUpdate | No (`BadHistoryOperationUnsupported`) | Yes (Insert / Replace / Update / DeleteRaw / DeleteAtTime) | Yes — soft delete (tombstones), see [below](#historyupdate-soft-delete) |
|
||||
| Modified-value history (`IsReadModified`, #668) | No (`BadHistoryOperationUnsupported`) | Yes — `SampleModified`, atomic with the change | Yes — `{prefix}sample_modified`, written before the change (orphan-on-crash) |
|
||||
| Alarm history | gateway `SendEvent` | `Event` table (insert-if-absent) | `{prefix}event` table (dedup upsert on `(ts, event_id)`) |
|
||||
| Provisioning (`EnsureTags`) | numeric types only; others skipped | `MERGE` upsert of every request, all types | Upsert into `{prefix}tag`, all types |
|
||||
| Connection | `Endpoint` + `ApiKey` + TLS keys (top-level `ServerHistorian:*`) | `ServerHistorian:SqlServer:*` | `ServerHistorian:QuestDb:*` |
|
||||
@@ -110,6 +111,7 @@ and all HistoryRead calls on historized nodes return `GoodNoData` (empty, not an
|
||||
"HistoryReadBatchParallelism": 4,
|
||||
"MaxConcurrentHistoryReadBatches": 16,
|
||||
"HistoryReadDeadline": "00:01:00",
|
||||
"MaxTimeWeightedRawRows": 1000000,
|
||||
"SqlServer": {
|
||||
"ConnectionString": "",
|
||||
"Schema": "hist",
|
||||
@@ -141,6 +143,7 @@ and all HistoryRead calls on historized nodes return `GoodNoData` (empty, not an
|
||||
| `MaxTieClusterOverfetch` | int | `65536` | (Node-manager paging; all backends — a `SqlServer` PK rules out tie clusters, so it is moot there.) Maximum samples the server will fetch in one shot to page through a tie cluster (multiple samples sharing one `SourceTimestamp`). A cluster larger than this ceiling fails `BadHistoryOperationUnsupported`. Raise to handle abnormally large tie clusters; the default covers all normal-data cases. |
|
||||
| `HistoryReadBatchParallelism` | int | `4` | Max historized nodes served **concurrently** within one HistoryRead batch (arch-review 03/S3). The SDK's `HistoryRead*` override surface is synchronous, so each arm bridges once at its boundary; this turns per-node serving from sequential (`N × CallTimeout` worst case) into `⌈N / P⌉ × CallTimeout`. Also caps the gateway-load multiplication a single client can command. `≤ 1` serves sequentially. |
|
||||
| `MaxConcurrentHistoryReadBatches` | int | `16` | Process-wide limit on HistoryRead (and HistoryUpdate) batches served **concurrently** (arch-review 03/S3). A flood of history clients degrades to `BadTooManyOperations` rather than exhausting the SDK request-thread pool. With `HistoryReadBatchParallelism` this caps total in-flight gateway reads at `MaxConcurrentHistoryReadBatches × HistoryReadBatchParallelism` (default 16 × 4 = 64). A batch that cannot get a slot within `min(HistoryReadDeadline, 5 s)` fails every handle `BadTooManyOperations`. |
|
||||
| `MaxTimeWeightedRawRows` | int | `1000000` | `SqlServer` / `QuestDb` only (#671). The most raw in-window samples one [time-weighted aggregate](#time-weighted-aggregates-part-13-671) read (TimeAverage, TimeAverage2, **Total**, Total2, Interpolative) may stream into the server-side evaluator — that family is **not** pushed down to SQL. The window query is capped at this + 1 rows; one row past the cap fails **that node** with `BadTooManyOperations` (not counted as a backend fault) instead of holding a pooled connection until `HistoryReadDeadline` and returning `BadTimeout`. Bound rows outside the window do not count; per-sample aggregates are not capped. 1 000 000 ≈ 11.5 days of a 1 Hz tag. Non-positive = unbounded (a `Validate()` warning). |
|
||||
| `HistoryReadDeadline` | TimeSpan | `00:01:00` | Per-request wall-clock deadline for a HistoryRead (and HistoryUpdate) batch (arch-review 03/S3). A node still in flight at the deadline surfaces `BadTimeout`, so one request can never hold an SDK request thread longer than this regardless of node count. Non-positive = unbounded (a `Validate()` warning). |
|
||||
|
||||
**`ServerHistorian:SqlServer`** (read only when `Backend=SqlServer`; bound by `HistorianBackendRegistry.BindSqlServer`):
|
||||
@@ -151,6 +154,9 @@ and all HistoryRead calls on historized nodes return `GoodNoData` (empty, not an
|
||||
| `SqlServer:Schema` | string | `hist` | Must match `^[A-Za-z_][A-Za-z0-9_]{0,127}$` (it is spliced into DDL/DML) | SQL schema that holds `SchemaVersion`, `Tag`, `Sample`, `Event` and the `SampleRow`/`TimeRow` table types. |
|
||||
| `SqlServer:CommandTimeout` | TimeSpan | `00:00:30` | Must be `> 0` | Per-command timeout, rounded up to whole seconds (minimum 1 s). The SQL backend does **not** use `CallTimeout`. |
|
||||
| `SqlServer:AutoCreateSchema` | bool | `true` | — | `true`: on first use apply any missing embedded script under a `sp_getapplock` (`OtOpcUaHistorianSchema`), so concurrent nodes serialise. `false`: only verify `SchemaVersion >= 1`; if behind, **every** historian call fails with a clear error naming the missing version (surfaced in the read client's health `LastError`, never a host crash). A failed bootstrap is retried on the next call, not latched. |
|
||||
| `SqlServer:RetentionDays` | int? | unset | `>= 1` when set | Days of history kept, on **source time**. Unset keeps everything. When set, driver nodes purge older `hist.Sample` / `hist.Event` rows — see [Retention](#retention-retentiondays). |
|
||||
| `SqlServer:PurgeInterval` | TimeSpan | `01:00:00` | `> 0` | How often the purge runs (first pass at start-up). |
|
||||
| `SqlServer:PurgeBatchSize` | int | `4000` | `>= 1` | Rows per `DELETE TOP (n)`; the default stays under SQL Server's ~5000-lock escalation threshold so a purge never takes a table lock. |
|
||||
|
||||
**`ServerHistorian:QuestDb`** (read only when `Backend=QuestDb`; bound by `HistorianBackendRegistry.BindQuestDb`): six keys — `ConnectionString`
|
||||
(required), `TablePrefix`, `RetentionDays`, `CommandTimeout`, `AutoCreateSchema`, `VisibilityTimeout`. The full table, defaults and validation
|
||||
@@ -242,6 +248,7 @@ and is registered the same way for every backend.
|
||||
| `DrainConcurrency` | int | `8` | Historian tags a single drain pass writes **concurrently**. One peeked batch is grouped per historian tag and each group is an independent gateway round-trip, so a sequential pass cost `tags × latency`; this window makes it `⌈tags / concurrency⌉ × latency` while bounding how many round-trips can be outstanding against the gateway at once. `1` restores the fully sequential drain; a value `< 1` is clamped to 1 (with a `Validate()` warning). **The ack is unchanged** — still all-or-nothing after every group has reported, and the #538 per-tag quarantine still keys on "another group wrote this pass". |
|
||||
| `Capacity` | int | `500000` | Max un-acked outbox entries before the drop-oldest policy sheds the front of the queue; `0` = unbounded. Roughly 50 MB of FasterLog. The #538 **disk backstop** for the one case the per-tag quarantine deliberately cannot resolve — a historian unreachable or refusing *every* write, where nothing proves any particular tag is at fault so the recorder correctly retains. |
|
||||
| `MaxWriteAttempts` | int | `5` | #538 poison-tag budget: consecutive failed drain passes one historian tag may accumulate before its queued entries are abandoned and the FIFO head is allowed to advance. **Counted only across passes where *other* tags wrote successfully**, so an ordinary gateway outage never advances it. Clamped to at least 1; each abandonment logs a Warning naming the tag. |
|
||||
| `CaptureWhilePrimaryUnreachable` | bool | `true` | #672: a standby whose capture is stood down for its mesh-partner Primary (#537) resumes capturing as soon as the cluster failure detector reports that Primary **unreachable**, instead of waiting for auto-down + a fresh redundancy snapshot. Shrinks a crash-failover capture gap from ~25 s to the failure-detector window (~10–12 s), at the cost of short duplicate windows — see [the crash-failover note](#a-crashed-primary-capture-gap-672). `false` restores the pre-#672 behaviour. Never affects the alarm drain or the Primary data-plane gate. |
|
||||
| `MinBackoffSeconds` | double | `1` | Initial retry backoff after a failed drain pass. |
|
||||
| `MaxBackoffSeconds` | double | `30` | Cap on the exponential retry backoff. |
|
||||
|
||||
@@ -381,6 +388,37 @@ trend is not. The residual cost is a capture gap across a failover, bounded by h
|
||||
node takes to see its first redundancy snapshot — inherent to any non-duplicating design. Suppressed
|
||||
values are counted in `RecorderStatus.SuppressedNotPrimary`, and each gate transition logs once.
|
||||
|
||||
### A crashed Primary: capture gap (#672)
|
||||
|
||||
On a **crash** (SIGKILL / `docker kill`) that "first snapshot" can be a long way off. The dead Primary
|
||||
stays a cluster member — *unreachable, not downed* — for the failure-detector window plus
|
||||
`auto-down-unreachable-after` (`acceptable-heartbeat-pause` 10 s + 15 s in `akka.conf`), and the
|
||||
redundancy singleton almost always lived on it (it is the oldest member), so no snapshot is published
|
||||
until auto-down re-homes the singleton. The standby's cached snapshot keeps naming the dead node Primary
|
||||
and **nobody captures** for ~30 s (measured on the QuestDB live gate).
|
||||
|
||||
`DriverHostActor` therefore follows cluster reachability (its own `ClusterEvent.ReachabilityEvent`
|
||||
subscription) and, with `ContinuousHistorization:CaptureWhilePrimaryUnreachable` on (the default),
|
||||
**excludes an unreachable peer from the historization "a mesh-partner Primary is recording" question**,
|
||||
re-deriving and republishing only the capture verdict from the last snapshot. A node at the same address
|
||||
coming Up again clears the mark; a reachable-again Primary closes the gate again. It logs
|
||||
`capturing historized values while recording Primary <id> is unreachable` once on entry and once on
|
||||
revert. The alarm drain and the Primary data-plane verdict (writes, native acks, the Sparkplug host claim)
|
||||
are **not** touched — a false allow there is a dual claimant, not a duplicate sample.
|
||||
|
||||
Trade-offs, all in the duplicate direction #537 already accepts:
|
||||
|
||||
- **The gap shrinks to the failure-detector window** (~10–12 s), not to zero — detection is the floor.
|
||||
- **Duplicates during the overlap.** A Primary that is only *briefly* unreachable (GC pause, network blip)
|
||||
overlaps the standby until it is reachable again, and under a genuine **partition** both sides capture
|
||||
from detection onward (as they already did after auto-down, now ~15 s earlier). SqlServer and QuestDB
|
||||
absorb a duplicate of a **source-timestamped** sample idempotently (`(TagId, SourceTimeUtc)` key /
|
||||
dedup), but a driver with **no source timestamp** (e.g. Modbus: `SourceTimestampUtc` is null, so each
|
||||
node stamps its own poll time) writes a duplicate row on **every** backend, and the **Gateway** backend
|
||||
stores duplicates of everything.
|
||||
- Set `CaptureWhilePrimaryUnreachable=false` to keep the pre-#672 behaviour (no duplicates from this
|
||||
path, the full ~30 s hole on a crash).
|
||||
|
||||
---
|
||||
|
||||
## Tag auto-provisioning (EnsureTags)
|
||||
@@ -480,9 +518,10 @@ The server supports all four OPC UA HistoryRead variants:
|
||||
| Variant | Node type | CLI `--aggregate` |
|
||||
|---|---|---|
|
||||
| **Raw** | Historized variable | (omit `--aggregate`) |
|
||||
| **Processed** | Historized variable | `Average`, `Minimum`, `Maximum`, `Count`, `Start`, `End`, `StandardDeviation` (the CLI's own list; `SqlServer` serves thirteen aggregates — see the [aggregate table](#aggregates-processed-reads)) |
|
||||
| **Processed** | Historized variable | `Average`, `Minimum`, `Maximum`, `Count`, `Start`, `End`, `StandardDeviation` (the CLI's own list; `SqlServer` / `QuestDb` serve seventeen aggregates — see the [aggregate table](#aggregates-processed-reads)) |
|
||||
| **AtTime** | Historized variable | n/a (client supplies exact timestamps) |
|
||||
| **Events** | Equipment folder (event notifier) | n/a |
|
||||
| **Modified** (Raw with `IsReadModified = true`, #668) | Historized variable | `--modified` (see [Modified-value history](#modified-value-history-part-11)) |
|
||||
|
||||
**Variable nodes** (historized tags) serve Raw, Processed, and AtTime history.
|
||||
`Historizing=true` and `AccessLevels.HistoryRead` are set at materialization so any compliant
|
||||
@@ -583,6 +622,76 @@ are disposed when the session closes). Resuming an unknown / evicted / released
|
||||
`BadContinuationPointInvalid`. `releaseContinuationPoints` drops the stored cursors without reading
|
||||
data.
|
||||
|
||||
### Continuation-point paging (Events) (#669)
|
||||
|
||||
A capped event read (`NumValuesPerNode > 0`) pages exactly like Raw, with one difference: the trigger is the **source's
|
||||
truncation flag** (`HistoricalEventsResult.ContinuationPoint != null`), never the page count, so a cap-sized page the source
|
||||
says is complete carries no continuation point. `NumValuesPerNode == 0` stays unbounded and never pages.
|
||||
|
||||
- The cursor is `(source, next start, end, boundary skip, cap)` in the same session-bound store as Raw (flagged as an events
|
||||
cursor — a Raw point presented to an events read, or the reverse, is `BadContinuationPointInvalid`). The next page reads from
|
||||
the last event's `EventTimeUtc` **inclusive** and drops the events at that timestamp already emitted.
|
||||
- A tie cluster larger than the cap is paged within by over-fetching the half-open window `[T, T + resolution)` up to
|
||||
`MaxTieClusterOverfetch`; beyond that bound the node answers `BadHistoryOperationUnsupported` (raise the bound).
|
||||
- **Tie exactness depends on the backend.** SqlServer and QuestDB order by `(EventTimeUtc, EventId)`, so the skip count is exact.
|
||||
The Gateway backend has no stable intra-timestamp order (and only a truncation flag), so ties at a page boundary are
|
||||
best-effort there.
|
||||
- Reverse event reads (`start > end`) are not paged server-side; no continuation point is issued for them.
|
||||
- No session on the request (in-process callers) degrades to single-shot, as for Raw.
|
||||
|
||||
### Modified-value history (Part 11)
|
||||
|
||||
**Gitea #668.** On the `SqlServer` and `QuestDb` backends every HistoryUpdate leaves a **modification trail**: one row per sample it
|
||||
inserted, replaced or deleted, carrying the value **as it was before** the change (none for an insert), the caller's principal, the
|
||||
modification time and the Part 11 `HistoryUpdateType`. A HistoryReadRaw with `IsReadModified = true` returns that trail as a
|
||||
`HistoryModifiedData` — one `DataValue` (the prior value) plus one parallel `ModificationInfo` (`ModificationTime`, `UpdateType`,
|
||||
`UserName`) per entry.
|
||||
|
||||
| Operation | Recorded `UpdateType` | Prior value |
|
||||
|---|---|---|
|
||||
| Insert | `Insert` | none — served `null` / `BadNoData` |
|
||||
| Replace | `Replace` | the replaced value and its status |
|
||||
| Update (upsert) | `Update` | the replaced value, or none when it inserted |
|
||||
| DeleteAtTime / DeleteRaw | `Delete` | the deleted value and its status |
|
||||
|
||||
- **Only applied items are recorded.** A refused item (`BadEntryExists`, `BadNoEntryExists`, `BadInvalidTimestamp`, …), a DeleteRaw over an
|
||||
empty range, the **live recorder's writes** and the **retention purge** leave nothing.
|
||||
- **`UserName` is the session's principal** — the API key's `NodeAcl` principal, or `anonymous` for an admitted anonymous session (the
|
||||
key id is stored too, not served). Longer than 200 characters is truncated, never refused.
|
||||
- **Window and order.** Same windows as Raw: forward `[start, end)` ascending by `(SourceTime, ModifiedId)`, reverse (`start > end`)
|
||||
`(end, start]` descending, `start == end` the modifications at exactly that time; an unset bound with a cap pages open-ended like Raw.
|
||||
Several modifications of one timestamp appear in the order they were made.
|
||||
- **Paging.** `NumValuesPerNode > 0` pages with session-bound continuation points keyed on the last entry's `(SourceTime, ModifiedId)` —
|
||||
an exact, exclusive key, so there are no ties to over-fetch and no trailing empty page (the backend reads `cap + 1` rows to know).
|
||||
The cursor is bound to the node and to the modified read kind (a raw or events point is `BadContinuationPointInvalid`).
|
||||
`NumValuesPerNode == 0` is unbounded.
|
||||
- **Authorization** is the ordinary **HistoryRead** decision — no new operation or permission (ungated with `Security:Acl:Enabled` off).
|
||||
- **Unsupported** ⇒ `BadHistoryOperationUnsupported`: the `Gateway` backend (the AVEVA Historian exposes no modification trail) and the
|
||||
`Null` default. **Deleting the trail** (`DeleteRawModifiedDetails.IsDeleteModified = true`) is not supported either — the trail is an
|
||||
audit record and is aged out only by retention (below).
|
||||
- **Atomicity differs by backend.** `SqlServer` writes the trail row with `OUTPUT … INTO` in the **same statement** that changes `hist.Sample`,
|
||||
so a change and its trail row commit or roll back together. QuestDB has no multi-table transaction, so the trail row is written **before**
|
||||
the change: a crash (or failed sample INSERT) in between leaves an **orphan trail row** describing a change that never landed — never a
|
||||
change without its trail row. Treat a QuestDB trail entry whose sample still shows the prior value as such an orphan.
|
||||
A failed sample write after its trail rows landed logs a **Warning** naming the tag, the row count and the time range, so the orphan
|
||||
can be found.
|
||||
- **Rolling upgrade (SqlServer).** The v2 schema is applied by the first node that upgrades. Until its peer is upgraded too, that
|
||||
not-yet-upgraded node still runs the old build, whose HistoryUpdates write **no trail rows** — silent gaps in the audit trail. Upgrade both
|
||||
nodes of a pair together (stop, upgrade, start both); modifications made by an old-build node in the window are never recorded.
|
||||
- **Paging vs a concurrent HistoryUpdate (SqlServer).** `ModifiedId` is an `IDENTITY`, which orders by allocation, not by commit. A modified
|
||||
read paging at the same moment a HistoryUpdate commits a modification of the **same source timestamp** can have already passed that
|
||||
row's id and so never see it on this pass (a fresh read does). The cursor is exact for a quiescent trail, not a snapshot guarantee.
|
||||
- **QuestDB `ModifiedId` across nodes.** Ids are UTC ticks bumped past the last one issued by that process, so a pair's ids interleave by
|
||||
each node's **wall clock**. Same-source-time modifications made on two nodes order by their clocks (clock skew can reorder them);
|
||||
within one node the order is strict.
|
||||
- **Retention.** The trail ages out on the same **source-time** cutoff as the samples it describes: `SqlServer` purges `hist.SampleModified`
|
||||
per tag alongside `hist.Sample`; QuestDB puts the same TTL on `{p}sample_modified`.
|
||||
|
||||
```bash
|
||||
otopcua-cli historyread -u opc.tcp://localhost:4840 -U apikey -P otua_<keyId>_<secret> \
|
||||
-n "ns=2;s=Plant/Modbus/dev1/Speed" --start 2026-01-15T00:00:00Z --end 2026-01-16T00:00:00Z --modified
|
||||
```
|
||||
|
||||
### Total aggregate
|
||||
|
||||
The OPC UA `Total` aggregate is **supported** over the HistorianGateway backend. The gateway exposes a
|
||||
@@ -597,9 +706,9 @@ buckets surface as `BadNoData`.
|
||||
`HistoryAggregateType` carries the wider Part 13 set (Interpolative, TimeAverage, Start, End, Delta, Range,
|
||||
Min/MaxActualTime, Start/EndBound, DeltaBounds, DurationGood/Bad, PercentGood/Bad, WorstQuality,
|
||||
NumberOfTransitions, AnnotationCount, StandardDeviationPopulation, VariancePopulation); `OtOpcUaNodeManager.MapAggregate`
|
||||
maps each standard aggregate NodeId. The **`SqlServer`** backend serves the thirteen listed in the
|
||||
[aggregate table](#aggregates-processed-reads); everything else (e.g. `Interpolative`, `TimeAverage`,
|
||||
`StandardDeviationPopulation`) is `BadAggregateNotSupported` for that node. The **HistorianGateway** backend only has retrieval modes for the original
|
||||
maps each standard aggregate NodeId (`TimeAverage2` and `Total2` were appended for #671). The **`SqlServer`** and
|
||||
**`QuestDb`** backends serve the seventeen listed in the [aggregate table](#aggregates-processed-reads); everything else
|
||||
(e.g. `StartBound`, `DurationGood`, `StandardDeviationPopulation`) is `BadAggregateNotSupported` for that node. The **HistorianGateway** backend only has retrieval modes for the original
|
||||
five (Average, Minimum, Maximum, Total, Count): for any other member `AggregateModeMapper` throws
|
||||
`HistoryAggregateNotSupportedException`, which the HistoryRead path turns into `BadAggregateNotSupported` for that
|
||||
node only (not a generic failure, and not counted as a backend fault). The OPC UA Client driver forwards every
|
||||
@@ -613,13 +722,9 @@ member upstream. A NodeId outside the mapped set (e.g. `StandardDeviationSample`
|
||||
sample per requested timestamp, so the single-shot backend returns the complete result in one
|
||||
read and there is no "full page ⇒ maybe more" signal to page on. Returning the full result with
|
||||
no continuation point is spec-conformant.
|
||||
- **No modified-value history** (`HistoryReadModified`). Requests for modified values return
|
||||
`BadHistoryOperationUnsupported`. This is **infra-gated, not a server-code gap**: the HistorianGateway
|
||||
backend (`GatewayHistorianDataSource`) exposes only a current-value read path — there is no
|
||||
modified/edited-history surface to source the data from. The server-side override is in place (it cleanly
|
||||
rejects modified reads per node) and `IsReadModified` is honoured; serving real modified-value history is
|
||||
unblocked only once the gateway grows a modified-read RPC. Until then, rejecting is the correct,
|
||||
spec-conformant behaviour.
|
||||
- **No modified-value history on the `Gateway` backend.** Modified reads (`IsReadModified = true`) are served by
|
||||
`SqlServer` and `QuestDb` ([Modified-value history](#modified-value-history-part-11), #668); the HistorianGateway exposes no
|
||||
modified/edited-history surface, so on `Gateway` (and with no historian) they return `BadHistoryOperationUnsupported`.
|
||||
|
||||
### Redundancy and authorization
|
||||
|
||||
@@ -640,15 +745,18 @@ bootstrapper and one tag catalog, so the schema is ensured once per process and
|
||||
|
||||
### Schema
|
||||
|
||||
Created by the embedded idempotent script `Schema/001_initial.sql` (mirrored at
|
||||
[`docs/historian/sqlserver/001_initial.sql`](historian/sqlserver/001_initial.sql)); `$(Schema)` is the configured schema.
|
||||
Created by the embedded idempotent scripts `Schema/001_initial.sql` and `Schema/002_sample_modified.sql` (mirrored at
|
||||
[`docs/historian/sqlserver/001_initial.sql`](historian/sqlserver/001_initial.sql) and
|
||||
[`docs/historian/sqlserver/002_sample_modified.sql`](historian/sqlserver/002_sample_modified.sql)); `$(Schema)` is the configured schema.
|
||||
The bootstrapper applies every script above the recorded version, in order, under the schema applock.
|
||||
|
||||
| Object | Purpose |
|
||||
|---|---|
|
||||
| `SchemaVersion(Version, AppliedUtc)` | Applied-script ledger. Required version: **1**. |
|
||||
| `SchemaVersion(Version, AppliedUtc)` | Applied-script ledger. Required version: **2** (since #668). |
|
||||
| `Tag(TagId int identity, Name nvarchar(400) unique, DataType tinyint, EngineeringUnit, Description, CreatedUtc, UpdatedUtc)` | One row per historian tagname. `DataType` is a `DriverDataType`. |
|
||||
| `Sample(TagId, SourceTimeUtc datetime2(7), ValueKind tinyint, NumValue float, IntValue bigint, StrValue nvarchar(max), StatusCode int, ServerTimeUtc)` | **PK `(TagId, SourceTimeUtc)`** — one value per timestamp (Part 11), which makes redelivery idempotent and rules out tie clusters. `StatusCode` is the uint status stored bit-for-bit. |
|
||||
| `Event(EventId nvarchar(64) PK nonclustered, SourceName, AlarmId, AlarmName, AlarmTypeName, Severity, EventKind, Message, UserName, Comment, EventTimeUtc, ReceivedTimeUtc)` | Alarm history. Clustered on `(EventTimeUtc, EventId)`; extra index `(SourceName, EventTimeUtc)`. |
|
||||
| `SampleModified(TagId, SourceTimeUtc, ModifiedId bigint identity, ModificationTimeUtc, UpdateType tinyint, Principal nvarchar(200), ApiKeyId, ValueKind, NumValue, IntValue, StrValue, StatusCode, ServerTimeUtc)` (v2) | The Part 11 [modification trail](#modified-value-history-part-11): one row per sample a HistoryUpdate changed, carrying the **prior** value columns (all NULL for an insert). Clustered PK `(TagId, SourceTimeUtc, ModifiedId)`. ⚠️ No FOREIGN KEY, CHECK or trigger — it is an `OUTPUT … INTO` target, which forbids them; adding one fails every HistoryUpdate. |
|
||||
| `SampleRow`, `TimeRow` table types | TVPs for batched sample writes/merges, AtTime and delete-at-time. |
|
||||
|
||||
Column mapping per `ValueKind`: `Double` → `NumValue`; `Int64` → `IntValue` + `NumValue` (as a double, for aggregates);
|
||||
@@ -656,9 +764,15 @@ Column mapping per `ValueKind`: `Double` → `NumValue`; `Int64` → `IntValue`
|
||||
`ValueKind` codes: Empty 0, Double 1, Int64 2, Boolean 3, String 4, DateTime 5. Timestamps are bound as `datetime2(7)`
|
||||
everywhere (full 100 ns precision — never a `DataTable`, which would round to 3.33 ms).
|
||||
|
||||
**Applying the script by hand** (needed when `AutoCreateSchema=false`): create the database, then
|
||||
`sqlcmd -S <server> -U <user> -P <pw> -d <db> -I -v Schema=hist -i docs/historian/sqlserver/001_initial.sql`
|
||||
(`-v Schema=` supplies the `$(Schema)` placeholder; verified against the docker-dev SQL container). Re-running is a no-op.
|
||||
**Applying the scripts by hand** (needed when `AutoCreateSchema=false`): create the database, then run each script in order —
|
||||
`sqlcmd -S <server> -U <user> -P <pw> -d <db> -I -v Schema=hist -i docs/historian/sqlserver/001_initial.sql`, then the same with
|
||||
`002_sample_modified.sql` (`-v Schema=` supplies the `$(Schema)` placeholder; verified against the docker-dev SQL container). Re-running is a no-op.
|
||||
|
||||
⚠️ **Upgrading an existing v1 historian (#668).** A build with this change requires schema **version 2**. With `AutoCreateSchema=true`
|
||||
(the default) the first historian call after the upgrade applies `002_sample_modified.sql` in place (it only creates `SampleModified`; existing
|
||||
samples are untouched). With `AutoCreateSchema=false` **every historian call fails** (`missing version 2 (found 1)`, in health `LastError`)
|
||||
until an operator runs `002_sample_modified.sql` by hand as above — do that before rolling the new build out. Modifications made before the
|
||||
upgrade have no trail; the trail starts with the first HistoryUpdate after it.
|
||||
|
||||
### Writes
|
||||
|
||||
@@ -725,15 +839,26 @@ result never has both.
|
||||
|
||||
`[start, end)` by `EventTimeUtc`; **a reversed window (`end < start`) is swapped and returned ascending**. A null `sourceName` means all
|
||||
sources, otherwise `SourceName = @source`. Ordered `(EventTimeUtc, EventId)` ascending. A bounded read fetches `maxEvents + 1` rows and
|
||||
returns a non-null continuation point when the extra row exists; the node manager does not currently issue or resume event
|
||||
continuation points, so a client-capped read returns only the first page.
|
||||
returns a non-null continuation point when the extra row exists; the node manager turns that flag into an OPC UA continuation
|
||||
point (see [Continuation-point paging (Events)](#continuation-point-paging-events-669)).
|
||||
|
||||
### Aggregates (Processed reads)
|
||||
|
||||
> **These are simple per-sample definitions, not full OPC UA Part 13.** There is no time-weighting, no interpolated interval-boundary
|
||||
> value and no partial-interval status. They are pushed down to T-SQL and suit trend-style consumption. Any `HistoryAggregateType` not
|
||||
> in the table throws `HistoryAggregateNotSupportedException` ⇒ `BadAggregateNotSupported` for that node (e.g. `Interpolative`,
|
||||
> `TimeAverage`, `StartBound`, `DurationGood`, `StandardDeviationPopulation`).
|
||||
> **Two kinds.** The *per-sample* aggregates (Average, Minimum, Maximum, Count, Range, Start/End/Delta, Min/MaxActualTime,
|
||||
> PercentGood/Bad) are simple per-sample definitions — no time-weighting, no interval-boundary values — pushed down to SQL. The
|
||||
> *time-weighted* aggregates (TimeAverage, TimeAverage2, Total, Total2, Interpolative; #671) follow OPC UA Part 13 with bounding values
|
||||
> — see [Time-weighted aggregates](#time-weighted-aggregates-part-13-671). Any `HistoryAggregateType` not in the tables throws
|
||||
> `HistoryAggregateNotSupportedException` ⇒ `BadAggregateNotSupported` for that node (e.g. `StartBound`, `DurationGood`,
|
||||
> `StandardDeviationPopulation`).
|
||||
>
|
||||
> ⚠️ **Release note / migration (#671): `Total` changed meaning on the `SqlServer` and `QuestDb` backends.** It was the plain `SUM` of
|
||||
> the usable samples in the interval, pushed down to SQL; it is now the Part 13 time-weighted integral, `TimeAverage × interval seconds`
|
||||
> (value·seconds), **evaluated in C# from the raw rows** (not pushed down) and therefore subject to `ServerHistorian:MaxTimeWeightedRawRows`
|
||||
> — a long `Total` over a dense tag that used to succeed may now fail `BadTooManyOperations`; read it in shorter windows. This brings
|
||||
> `Total` closer to the `Gateway` backend, whose `Total` maps to AVEVA's native time-weighted `Integral` retrieval mode — but AVEVA's
|
||||
> integral time unit/divisor has not been verified, so the two backends may differ by a constant factor for the same data. A client
|
||||
> that relied on the old sum gets a different number for the same data (e.g. two samples 10 and 20 ten seconds apart no longer total
|
||||
> 30). There is no switch back; `Count` × `Average` reproduces the old sum.
|
||||
|
||||
**Bucketing.** One result per interval of `[start, end)`; bucket *i* = `[start + i·interval, start + (i+1)·interval)`, start-inclusive /
|
||||
end-exclusive to the tick, the last bucket truncated at `end`. `interval <= 0` ⇒ a single interval `[start, end)`; `start == end` ⇒ no
|
||||
@@ -749,7 +874,7 @@ status carries the historian *Calculated* info bit, `BadNoData` included.
|
||||
|
||||
| Aggregate | Computed over | Value timestamp |
|
||||
|---|---|---|
|
||||
| `Average`, `Minimum`, `Maximum`, `Total` | `AVG` / `MIN` / `MAX` / `SUM` of `NumValue` over usable samples | bucket start |
|
||||
| `Average`, `Minimum`, `Maximum` | `AVG` / `MIN` / `MAX` of `NumValue` over usable samples | bucket start |
|
||||
| `Count` | number of usable samples (`Int32`) | bucket start |
|
||||
| `Range` | `Maximum − Minimum` | bucket start |
|
||||
| `Start`, `End` | first / last usable sample by time (keeps the sample's type) | bucket start |
|
||||
@@ -766,12 +891,118 @@ Status and edge rules:
|
||||
- **`PercentGood` / `PercentBad` and Uncertain samples:** Uncertain samples are counted in the denominator but are **neither Good nor Bad**, so `PercentGood + PercentBad < 100` whenever a bucket holds Uncertain samples. Both report status `Good` for any non-empty bucket.
|
||||
- Everything except `Count` and `Start`/`End` is a `double`; `Int64` samples aggregate as doubles and `Boolean` as 0/1.
|
||||
|
||||
### Time-weighted aggregates (Part 13, #671)
|
||||
|
||||
`TimeAverage`, `TimeAverage2`, `Total`, `Total2` and `Interpolative`, evaluated by the shared
|
||||
`Driver.Historian.Sql.Common/TimeWeightedAggregator.cs` for **both** SQL backends (so `SqlServer` and `QuestDb` agree exactly).
|
||||
**This whole family — `Total` included — is evaluated in C#, not pushed down to SQL** (see *Pushdown trade-off* and the raw-row cap
|
||||
below). Bucketing (edges, truncated last bucket, the 100 000-interval cap) is the same as above; reverse windows are covered below.
|
||||
|
||||
**Server defaults, fixed:** `TreatUncertainAsBad = false`, `PercentDataBad = PercentDataGood = 100`, `UseSlopedExtrapolation = false`.
|
||||
**A client's `AggregateConfiguration` in the HistoryRead request is ignored** (the node manager does not forward it); every read uses
|
||||
these defaults.
|
||||
|
||||
**Bounding values.** Besides the window `[start, end)`, each read fetches four single-row seeks: the latest *usable* sample before
|
||||
`start`, the latest sample of any quality before `start` (they differ when an outage straddles the start), the first sample of any
|
||||
quality at or after `end`, and the first *usable* one at or after `end`. So a bucket with no samples of its own, between two samples,
|
||||
still gets a value.
|
||||
|
||||
**The signal.** Between a usable sample `p` and the next sample `s`: both `Double`/`Int64` ⇒ **sloped** (linear), segment quality the
|
||||
worse of the two; either `Boolean` ⇒ **stepped** (`p` held), quality `p`'s; `s` Bad ⇒ `p` held up to the outage (never interpolated into
|
||||
it — the AtTime rule), Uncertain; no later sample at all ⇒ `p` held to the window end (stepped extrapolation), Uncertain. From a Bad
|
||||
sample (incl. an `Empty` outage marker) to the next sample the signal is **Bad** (outage); before the first sample there is **no
|
||||
data**. That is the
|
||||
*simple-bounding* signal. The *interpolated-bounding* signal skips Bad samples: it joins consecutive usable samples (sloped / stepped)
|
||||
straight across an outage.
|
||||
|
||||
| Aggregate | Value | Bad intervals |
|
||||
|---|---|---|
|
||||
| `TimeAverage` | ∫ interpolated-bounding signal ÷ its defined duration | interpolated across (Part 13) |
|
||||
| `Total` | `TimeAverage × interval seconds` (value·s; the truncated last bucket uses its own length) | interpolated across |
|
||||
| `TimeAverage2` | ∫ simple-bounding signal over non-Bad time ÷ non-Bad duration | excluded |
|
||||
| `Total2` | ∫ simple-bounding signal over non-Bad time (value·s) | excluded |
|
||||
| `Interpolative` | interpolated-bounding value **at the interval start** (reverse read: see below) | interpolated across |
|
||||
|
||||
**Status, the four time-weighted values** — aligned with the OPC Foundation SDK's `AggregateCalculator.GetTimeBasedStatusCode`
|
||||
(inspected in `Opc.Ua.Server` 1.5.378.106) under the fixed defaults. With `TreatUncertainAsBad = false`, **Uncertain time counts as
|
||||
Good time**: Uncertain samples, a value held up to an outage and the stepped extrapolation past the last sample all count toward
|
||||
PercentDataGood.
|
||||
|
||||
- Every instant of the interval non-Bad ⇒ `Good`.
|
||||
- Some non-Bad time but not all (an outage, or time before the first sample) ⇒ `UncertainDataSubNormal`.
|
||||
- No non-Bad time but some outage time ⇒ generic **`Bad`** (`0x80000000`) — the SDK's all-Bad result, value null.
|
||||
- No data at all ⇒ `BadNoData`.
|
||||
- A usable `String`/`DateTime` sample feeding the interval ⇒ `BadAggregateInvalidInputs`.
|
||||
|
||||
All carry the Calculated bit; a non-Bad result for the **truncated last interval** (shorter than the processing interval) also carries
|
||||
the **Partial** bit (`0x4`, `AggregateBits.Partial`). Timestamp = interval start; value `double` (Boolean 0/1). (The per-sample
|
||||
aggregates do not set Partial — a follow-up.)
|
||||
|
||||
> **A live tag's latest interval.** Past the tag's last stored sample the value is held (stepped extrapolation, Uncertain). Under
|
||||
> the SDK rule that Uncertain time is Good time, so the latest interval of a live tag normally reads **`Good`** — not Uncertain —
|
||||
> with the last value held to the interval end. It reads Uncertain only if the interval also contains outage time or starts
|
||||
> before the tag's first sample. Do not take a Good latest interval as proof that data arrived up to its end. If it was truncated
|
||||
> by the request's end time it also carries **Partial**.
|
||||
|
||||
**Status, `Interpolative`** (point-based, not the PercentDataGood rule):
|
||||
- A usable sample exactly at the point ⇒ that value with **its own stored status** (no historian bit).
|
||||
- No usable sample at or before the point ⇒ plain `BadNoData`.
|
||||
- Otherwise the value with the **Interpolated** bit:
|
||||
- `Good` on a segment whose defining samples are Good;
|
||||
- `UncertainDataSubNormal` inside or up to an outage (bridged) and past the last sample (extrapolated).
|
||||
|
||||
**Reverse reads (`end < start`).** Part 13 anchors a reverse read at the request's start, which is the later time. `Interpolative`
|
||||
follows that: its points are `start, start − interval, …` (in the forward window's terms `End − i·interval`), latest first. A
|
||||
reverse read 10:00 → 09:00 at 1 h returns the value at **10:00**, not 09:00, and no point ever falls before the window. The interval
|
||||
aggregates (TimeAverage, Total, …) keep the existing backend-wide reverse rule: computed forward with intervals anchored at the
|
||||
earlier time, returned in descending order. Re-anchoring those at the later time for every aggregate is a follow-up.
|
||||
|
||||
**Pushdown trade-off.** Unlike the per-sample aggregates, the time-weighted family is **not** pushed down to SQL: the backend streams
|
||||
the window's rows plus the four bounds (one batch of five result sets on SQL Server; a bounds `UNION ALL` + a window select on QuestDB)
|
||||
and the evaluator splits segments at bucket edges in C#. Network traffic is therefore O(samples in the window) rather than O(buckets);
|
||||
evaluation memory stays O(buckets) (rows are streamed, never materialised). The alternative — the same Part 13 segment
|
||||
interpolation, edge splitting and two bounding definitions written twice, in T-SQL and QuestDB SQL, and kept bit-identical — was
|
||||
judged the larger risk. For very long, densely sampled windows prefer the pushed-down `Average` (or a coarser read) over
|
||||
`TimeAverage`.
|
||||
|
||||
**Raw-row cap.** The window query is limited to `ServerHistorian:MaxTimeWeightedRawRows` + 1 rows (default 1 000 000, shared by both
|
||||
SQL backends; `TOP (@maxRows)` / `LIMIT $4`). When that extra row arrives the read stops and the node fails with
|
||||
**`BadTooManyOperations`** — for that node only; it is not counted in the backend's health failures. Without the cap a one-year
|
||||
`Total` on a 1 Hz tag (≈31 M rows) would hold a pooled connection until `HistoryReadDeadline` and then return `BadTimeout`. The four
|
||||
bound rows do not count toward the cap, and the per-sample aggregates are never capped. Raise the cap, or read in shorter windows.
|
||||
|
||||
### Health
|
||||
|
||||
The read client reports `GetHealthSnapshot()` (query/failure counters, consecutive failures, `LastError` — e.g. a missing schema under
|
||||
`AutoCreateSchema=false`). A SQL error on a read surfaces as `BadHistoryOperationUnsupported` for **that node only**; a cancelled read
|
||||
(deadline, client abort) is not counted as a backend fault.
|
||||
|
||||
### Retention (`RetentionDays`)
|
||||
|
||||
`hist.Sample`, `hist.SampleModified` and `hist.Event` grow without bound unless `ServerHistorian:SqlServer:RetentionDays` is set. QuestDB already has an
|
||||
equivalent TTL (`QuestDb:RetentionDays`); the Gateway backend's retention is the AVEVA Historian's own.
|
||||
|
||||
- **What runs.** `HistorianRetentionService` (a hosted service, driver-role nodes, Backend=`SqlServer` with `RetentionDays` set; inert
|
||||
otherwise) ticks every `PurgeInterval` (first pass at start-up) and calls the backend's `IHistorianRetention.PurgeAsync`.
|
||||
- **Gate vs. mutual exclusion.** Each tick consults `IHistorizationRoleView.ShouldRecordHistorizedValues` (the capture verdict; a
|
||||
standby skips, an unknown role runs). That verdict is **pair-scoped** while the purge is **database-scoped**, so the gate is only a
|
||||
load reducer: the `sp_getapplock` below is the actual mutual exclusion. Pairs sharing one historian database, and both nodes of a
|
||||
pair on their boot-time pass, simply serialise on it. Set `RetentionDays` **uniformly on every node that shares a historian
|
||||
database** — if they differ, the smallest window wins.
|
||||
- **How it deletes.** Cutoff = `UtcNow - RetentionDays` on source time. Samples are purged **per tag** with
|
||||
`DELETE TOP (@n) ... WHERE TagId = @id AND SourceTimeUtc < @cutoff` (a seek on the clustered `(TagId, SourceTimeUtc)` key; a global
|
||||
time predicate would scan) — and, in the same per-tag pass, the modification trail with
|
||||
`DELETE TOP (@n) FROM SampleModified WHERE TagId = @id AND SourceTimeUtc < @cutoff` (#668; same **source-time** cutoff, clustered seek; the
|
||||
purge itself never writes a trail row) — then events with `DELETE TOP (@n) ... WHERE EventTimeUtc < @cutoff` (seek on `CX_Event_Time`), each looped
|
||||
until a batch comes back short. Cancellation is honoured between batches. The `hist.Tag` catalogue is never purged.
|
||||
- **Shared databases.** Several pairs may point at one database; a session `sp_getapplock` (`OtOpcUa.Historian.Retention.<schema>`,
|
||||
timeout 0) lets one purger run and the others skip that tick (Debug log).
|
||||
- **Observability.** Counter `otopcua.historian.retention.purged{table=Sample|SampleModified|Event}`; a failed pass logs a Warning and retries next
|
||||
interval — it never affects capture or reads.
|
||||
- **Not done: partition switching.** For very large installs (billions of rows) a partitioned `hist.Sample` with sliding-window
|
||||
`SWITCH`/drop is cheaper than row deletes; the purger is deliberately simple batched deletes. Another table is added by one more
|
||||
`DeleteLoopAsync` call in `SqlServerHistorianRetention.PurgeAsync` (as `hist.SampleModified` was, #668).
|
||||
|
||||
### Docker-dev recipe
|
||||
|
||||
The docker-dev `sql` container (`otopcua-dev-sql-1`, host port `14330`, `sa` with the committed dev password from
|
||||
@@ -795,7 +1026,9 @@ sqlcmd -S localhost,14330 -U sa -P 'OtOpcUa!Dev123' -C -I -d OtOpcUaHistorian \
|
||||
-Q "SELECT t.Name, COUNT(*) n, MAX(s.SourceTimeUtc) last FROM hist.Sample s JOIN hist.Tag t ON t.TagId=s.TagId GROUP BY t.Name"
|
||||
```
|
||||
|
||||
Only the Primary of a pair records, so the row count advances once per published value, not twice. Read it back with
|
||||
Only the Primary of a pair records, so the row count advances once per published value, not twice. To watch retention, add
|
||||
`ServerHistorian__SqlServer__RetentionDays: "1"` and `ServerHistorian__SqlServer__PurgeInterval: "00:00:30"`, insert a `hist.Sample` row
|
||||
with an old `SourceTimeUtc` for an existing tag, and see it vanish within a tick while recent rows stay. Read it back with
|
||||
`Client.CLI historyread` and rewrite it with `Client.CLI historyupdate` ([Client.CLI.md](Client.CLI.md)). The password above is the
|
||||
committed docker-dev credential, not a secret; use `${secret:…}` / environment for anything real.
|
||||
|
||||
@@ -822,7 +1055,7 @@ the spike that pinned every QuestDB-specific form: [plans/2026-10-10-questdb-spi
|
||||
`questdb/questdb:10.0.1`.
|
||||
|
||||
QuestDB serves every value kind (`Double`, `Int64`, `Boolean`, `String`, `DateTime`, `Empty`), stores the real status, records comms-loss
|
||||
outage markers, serves the thirteen aggregates and supports **HistoryUpdate** (soft delete), exactly like `SqlServer`. The same conformance
|
||||
outage markers, serves the seventeen aggregates (time-weighted ones through the same shared evaluator, so results are identical) and supports **HistoryUpdate** (soft delete), exactly like `SqlServer`. The same conformance
|
||||
suite (`tests/Drivers/Historian.Conformance`) runs against both backends; the differences below are the only ones.
|
||||
|
||||
### Configuration (`ServerHistorian:QuestDb`)
|
||||
@@ -833,10 +1066,10 @@ Read only when `Backend=QuestDb`; bound by `HistorianBackendRegistry.BindQuestDb
|
||||
| Key | Default | Validation | Meaning |
|
||||
|---|---|---|---|
|
||||
| `QuestDb:ConnectionString` | `""` | **Required** (non-blank) | Npgsql connection string for QuestDB's PG-wire port, e.g. `Host=questdb;Port=8812;Username=admin;Password=quest;Database=qdb;ServerCompatibilityMode=NoTypeLoading`. `ServerCompatibilityMode=NoTypeLoading` skips the `pg_type` load; every query form was verified with it. Supply via env `ServerHistorian__QuestDb__ConnectionString` / `${secret:}` for anything real. |
|
||||
| `QuestDb:TablePrefix` | `hist_` | `^[A-Za-z0-9_]+$` | QuestDB has no schemas, so the three tables are `{prefix}sample`, `{prefix}event`, `{prefix}tag`. Spliced into DDL/DML, hence the regex. |
|
||||
| `QuestDb:RetentionDays` | unset | `>= 1` when set | Applied as table TTL to `{prefix}sample` and `{prefix}event` (see [Retention](#retention-ttl)). Unset keeps data forever. |
|
||||
| `QuestDb:TablePrefix` | `hist_` | `^[A-Za-z0-9_]+$` | QuestDB has no schemas, so the four tables are `{prefix}sample`, `{prefix}event`, `{prefix}tag` and `{prefix}sample_modified` (#668). Spliced into DDL/DML, hence the regex. |
|
||||
| `QuestDb:RetentionDays` | unset | `>= 1` when set | Applied as table TTL to `{prefix}sample`, `{prefix}event` and `{prefix}sample_modified` (see [Retention](#retention-ttl)). Unset keeps data forever. |
|
||||
| `QuestDb:CommandTimeout` | `00:00:30` | `> 0` | Per-command timeout, rounded up to whole seconds (minimum 1 s). |
|
||||
| `QuestDb:AutoCreateSchema` | `true` | — | `true`: create the tables (and reconcile the TTL) on first use. `false`: only verify the three tables exist; if one is missing every historian call fails with `HistorianSchemaMissingException` (health `LastError`, no crash) and no TTL is touched. A failed bootstrap is not latched: it is re-thrown for 5 s without touching QuestDB, then retried. |
|
||||
| `QuestDb:AutoCreateSchema` | `true` | — | `true`: create the tables (and reconcile the TTL) on first use. `false`: only verify the four tables exist (⚠️ an install from before #668 lacks `{prefix}sample_modified` — create it with the DDL in `QuestDbTables.CreateSampleModified` before upgrading, or every call fails); if one is missing every historian call fails with `HistorianSchemaMissingException` (health `LastError`, no crash) and no TTL is touched. A failed bootstrap is not latched: it is re-thrown for 5 s without touching QuestDB, then retried. |
|
||||
| `QuestDb:VisibilityTimeout` | `00:00:05` | `>= 0` | How long a write waits for its rows to become visible to queries ([WAL visibility](#wal-visibility-and-the-read-your-writes-wait)). `0` disables the wait. |
|
||||
|
||||
⚠️ **Connection pool size.** QuestDB's default limit is **64** concurrent PG-wire connections (`pg.net.connection.limit`), while Npgsql's
|
||||
@@ -848,14 +1081,15 @@ limit is server-configurable. The effective value is logged once at Information
|
||||
|
||||
### Tables
|
||||
|
||||
All three are **WAL tables with `DEDUP UPSERT KEYS`**, so re-sending a row with the same key replaces it (the upsert is what makes
|
||||
HistoryUpdate Replace work; the writer's own insert-if-absent semantics are built on top, below).
|
||||
All are **WAL tables**; the first three carry **`DEDUP UPSERT KEYS`**, so re-sending a row with the same key replaces it (the upsert is what
|
||||
makes HistoryUpdate Replace work; the writer's own insert-if-absent semantics are built on top, below). The modification trail is append-only.
|
||||
|
||||
| Table | Key | Partitioning |
|
||||
|---|---|---|
|
||||
| `{p}sample` (`ts, tag, kind, num_value, int_value, str_value, status, deleted, server_ts`) | `(ts, tag)` | `PARTITION BY DAY` (TTL when configured) |
|
||||
| `{p}event` (one-for-one with `hist.Event`) | `(ts, event_id)` | `PARTITION BY DAY` (TTL when configured) |
|
||||
| `{p}tag` (`ts, name, data_type, eng_unit, description, updated_utc`) | `(ts, name)` with `ts` fixed at `1970-01-01` — an upsert by name | `PARTITION BY YEAR`, never expires |
|
||||
| `{p}sample_modified` (`ts` = the sample's source time, `tag, modified_id, modification_ts, update_type, principal, api_key_id, kind, num_value, int_value, str_value, status, server_ts`) (#668) | none — append-only, no DEDUP; `modified_id` (process-monotonic ≈ UTC ticks) orders a timestamp's trail | `PARTITION BY DAY` (TTL when configured). A NULL `status` marks "no prior value" (QuestDB `BYTE` has no NULL, so `kind` is 0) |
|
||||
|
||||
`status` holds the uint OPC UA status code **unsigned** in a `LONG` (so "Bad" is `status >= 2147483648`, not `< 0`); `int_value` also holds
|
||||
Booleans as 0/1; `kind` is the `HistorianValueKind` in a `BYTE`. Tag names are limited to 400 characters (longer is a per-tag failure,
|
||||
@@ -922,6 +1156,11 @@ present **and not deleted**.
|
||||
| DeleteAtTime | write a tombstone (`deleted = true`, value NULL, `BadNoData`) where a live row exists | `Good` / `BadNoEntryExists` |
|
||||
| DeleteRaw | tombstone every live row in `[start, end)`, 1,000 per page, waiting for visibility between pages | the same codes as `SqlServer` |
|
||||
|
||||
**Modification trail (#668).** Each operation first appends one `{p}sample_modified` row per sample it will change, carrying the live row's
|
||||
current value (read by the same existence query), then writes the change. No multi-table transaction exists, so this order is the guarantee:
|
||||
a crash between the two leaves an **orphan trail row**, never an untracked change — the documented deviation from `SqlServer`'s atomic trail.
|
||||
See [Modified-value history](#modified-value-history-part-11).
|
||||
|
||||
The per-item codes come from the shared `HistoryUpdatePlanner`, so they equal `SqlServer`'s (including the duplicate-timestamp-in-one-request rules
|
||||
above). **Every read and aggregate filters `NOT deleted`.** A client Insert at a deleted timestamp re-populates it (the OPC UA intent); only the
|
||||
*live writer's* redelivery is barred from doing so.
|
||||
@@ -947,7 +1186,7 @@ Every query filters `NOT deleted`; every value goes through `StoredSample`, so t
|
||||
|
||||
### Retention (TTL)
|
||||
|
||||
`RetentionDays` becomes `TTL n DAYS` on `{p}sample` and `{p}event` (never on `{p}tag`: TTL must be a multiple of the partition size). QuestDB drops
|
||||
`RetentionDays` becomes `TTL n DAYS` on `{p}sample`, `{p}event` and `{p}sample_modified` (never on `{p}tag`: TTL must be a multiple of the partition size). QuestDB drops
|
||||
expired day partitions itself, tombstones included.
|
||||
|
||||
- At bootstrap the current TTL is read from `tables()` (`ttlValue`, `ttlUnit`). QuestDB **normalises the unit** (7 days reads `1 WEEK`), so the
|
||||
@@ -1011,7 +1250,8 @@ batch limiter and `HistoryReadDeadline`.
|
||||
| `DeleteAtTimeDetails` | `HistoryDeleteAtTime` | per time: `Good`; `BadNoEntryExists` |
|
||||
|
||||
Each backend call is applied as a unit (SqlServer: one transaction, `MERGE … OUTPUT $action` against the TVP; QuestDb: an INSERT per
|
||||
chunk plus the WAL-visibility wait, no transaction — see [QuestDB backend](#questdb-backend)). The node-level `StatusCode` stays `Good` when item results are
|
||||
chunk plus the WAL-visibility wait, no transaction — see [QuestDB backend](#questdb-backend)). Every applied item is also recorded, with the
|
||||
caller's principal, in the [modification trail](#modified-value-history-part-11) (#668). The node-level `StatusCode` stays `Good` when item results are
|
||||
mixed (Part 11 reports item outcomes in `OperationResults`); it is Bad only for whole-node failures.
|
||||
|
||||
**Per-item rejections** (answered without touching SQL; other items still proceed): a null item ⇒ `BadInvalidArgument`;
|
||||
@@ -1035,8 +1275,8 @@ outcome, later ones `BadNoEntryExists`. **Unknown tag:** Insert / Update create
|
||||
(DeleteRaw: `BadNoData`) without creating anything.
|
||||
|
||||
**Unsupported ⇒ `BadHistoryOperationUnsupported`** (per node): the node is not historized; the backend has no updater (`Gateway`, or
|
||||
`ServerHistorian:Enabled=false`); `PerformUpdateType.Remove`; `DeleteRawModifiedDetails.IsDeleteModified=true` (no modified-value history is
|
||||
kept); and the out-of-scope detail types — **event** HistoryUpdate (`UpdateEventDetails` / `DeleteEventDetails`) and `UpdateStructureData` —
|
||||
`ServerHistorian:Enabled=false`); `PerformUpdateType.Remove`; `DeleteRawModifiedDetails.IsDeleteModified=true` (the
|
||||
[modification trail](#modified-value-history-part-11) is an audit record, aged out only by retention); and the out-of-scope detail types — **event** HistoryUpdate (`UpdateEventDetails` / `DeleteEventDetails`) and `UpdateStructureData` —
|
||||
which are not overridden, so the SDK base rejects them. A backend exception is `BadInternalError` for that node and a deadline is `BadTimeout`.
|
||||
No `AuditHistoryUpdateEvent` is emitted.
|
||||
|
||||
|
||||
+10
-1
@@ -546,7 +546,16 @@ Under warm/hot redundancy both cluster nodes run `ScriptedAlarmHostActor` and ev
|
||||
node. ⚠️ **The peer it stands down for must record the same refs** — a Primary in its own per-cluster
|
||||
mesh — not merely hold the same rows; a node with no cluster-scoped role never stands down, because a
|
||||
fleet-wide election's Primary usually runs a different cluster's configuration and deferring to it would
|
||||
end this cluster's historization silently. See [Historian.md](Historian.md).
|
||||
end this cluster's historization silently. **Crash failover (Gitea #672):** a SIGKILLed Primary stays an
|
||||
unreachable-but-not-downed member until auto-down (10 s failure detector + 15 s), and the redundancy
|
||||
singleton usually died with it, so the standby's cached snapshot kept naming it Primary and nobody
|
||||
captured for ~30 s. `DriverHostActor` now follows `ClusterEvent.ReachabilityEvent` and excludes an
|
||||
**unreachable** peer from the historization peer-is-Primary question only (re-deriving the capture verdict
|
||||
from the cached snapshot), shrinking the gap to the failure-detector window; gated by
|
||||
`ContinuousHistorization:CaptureWhilePrimaryUnreachable` (default `true`). The alarm drain and the Primary
|
||||
data-plane verdict are **unchanged** — they still wait for a snapshot. The cost is duplicate capture while a
|
||||
live Primary is merely unreachable, and from detection onward under a real partition. See
|
||||
[Historian.md](Historian.md) §"A crashed Primary: capture gap (#672)".
|
||||
|
||||
Net effect: each alarm transition appears **once** on `/alerts` and would historize once, not once per node.
|
||||
|
||||
|
||||
+60
-11
@@ -45,22 +45,69 @@ out-of-process isolation.
|
||||
```
|
||||
|
||||
`EndpointUrls` (a list) takes precedence over the single-URL `EndpointUrl` and
|
||||
provides ordered **failover** — the driver tries each candidate in turn at init
|
||||
and on session drop, and the first to connect wins (e.g. a hot-standby pair on
|
||||
provides ordered **failover** — the driver tries each candidate in turn at every
|
||||
`InitializeAsync` (including the comms-loss watchdog's re-initialise), and the first to connect wins (e.g. a hot-standby pair on
|
||||
4840 / 4841). See
|
||||
`src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient.Contracts/OpcUaClientDriverOptions.cs`
|
||||
for every field (security policy/mode, auth type, session timeout, keep-alive,
|
||||
reconnect period, browse root, node/depth caps).
|
||||
browse root, node/depth caps). `ReconnectPeriod` is still parsed but no longer used (#670).
|
||||
|
||||
### Session lifetime
|
||||
|
||||
A single `Session` per driver instance; subscriptions multiplex onto it. The
|
||||
SDK reconnect handler takes the session down and brings it back on remote-server
|
||||
restart, re-sending subscriptions on reconnect so monitored-item handles don't
|
||||
dangle. Stored NodeIds embed the server-stable namespace **URI** (not the
|
||||
A single `Session` per driver instance; subscriptions multiplex onto it. Stored NodeIds embed the server-stable namespace **URI** (not the
|
||||
session-relative `ns=N` index) so a remote namespace-table reorder across a
|
||||
restart doesn't silently re-point references at the wrong namespace.
|
||||
|
||||
### Upstream outage and recovery (Gitea #670)
|
||||
|
||||
The session keep-alive (`KeepAliveInterval`, default 5 s) is the liveness signal:
|
||||
|
||||
- **Good keep-alive / any data or event notification** advances `DriverHealth.LastSuccessfulRead` — a
|
||||
subscription-only driver never calls `ReadAsync`, so without this its read stamp froze at connect time.
|
||||
- **Bad keep-alive** ⇒ health `Degraded` (`LastError = "keep-alive: <status>"`, read stamp preserved), host status
|
||||
`Stopped`, and — **once per outage**, not per keep-alive — every subscribed ref is published `BadNoCommunication`
|
||||
with its last value retained and **source + server timestamp = the detection time** (live-gate F1: stamped with the
|
||||
last Good's source time it collided with that row's `(TagId, SourceTimeUtc)` history key and was never stored), plus a quality-only `BadNoCommunication` annotation (#626 pattern, no alarm-state
|
||||
change, no `/alerts` row) on every subscribed alarm condition.
|
||||
- **The session survives** (a network blip; the keep-alive turns Good again) ⇒ recovered in place: health `Healthy`,
|
||||
the refs still Bad are re-read once and published Good (a server whose subscription survived does not resend an
|
||||
unchanged value, and a `(value, status)` dedup downstream lets the identical value through), alarm quality goes
|
||||
back Good and a `ConditionRefresh` re-reports anything missed. A notification that lands first wins over the read.
|
||||
For a bounded **replay window** after such a recovery — max(30 s, 2 x the longest subscription keep-alive window) —
|
||||
each ref is guarded by a high-water mark — its last Good upstream source time, raised by the recovery read and
|
||||
advanced by every notification accepted in the window — and a notification whose source time is at or BELOW it
|
||||
(equal included: OPC UA moves the source timestamp on every value or status change, so an equal one is a replay)
|
||||
is dropped: the SDK was seen delivering a missed one 4–6 s late, which regressed the node (live-gate F2). The
|
||||
guard holds for the WHOLE window — fresh notifications arrive ~1 s after recovery, before the late replay, so a
|
||||
ref is never disarmed early (the F2 re-gate failed on exactly that). A notification with no source time passes
|
||||
without moving the mark. Outside that window every notification is
|
||||
accepted whatever its timestamp, so an upstream clock that steps backwards (NTP, a PLC clock reset) never
|
||||
freezes a node. The outage Bad never moves the comparison; a re-initialise resets it.
|
||||
- **The session does not come back** (an upstream restart: the server forgot the session) ⇒ the driver stays
|
||||
`Degraded` with its read stamp frozen, which is exactly the #527 comms-loss watchdog's trip condition. After 90 s
|
||||
`DriverInstanceActor` runs `ShutdownAsync` + `InitializeAsync` (retried every 10 s while the server is down) and
|
||||
re-subscribes data + alarms; the new subscription's initial values come back Good. Alarm last-told state survives
|
||||
the re-initialise (#619), so the post-subscribe `ConditionRefresh` raises no un-ack flicker; the host's reconnect
|
||||
annotation restores condition quality.
|
||||
- **The subscription dies but the session lives** (an outage longer than the subscription lifetime —
|
||||
`LifetimeCount` x publishing interval, often well under the session timeout — or a server that drops it) ⇒ keep-alives
|
||||
stay Good, so they cannot be the signal. A `PublishError` of `BadNoSubscription` (while subscriptions are tracked) or
|
||||
`BadSubscriptionIdInvalid` naming a tracked subscription, or a tracked subscription whose SDK `PublishingStopped`
|
||||
persists for one more subscription keep-alive window + 5 s across good keep-alives, declares it lost: `Degraded`
|
||||
(`"subscription lost: …"`), the outage published once, and — sticky until the session is released — **nothing**
|
||||
advances `LastSuccessfulRead` any more (not good keep-alives, not a sibling subscription's notifications), so the
|
||||
watchdog's re-initialise recreates it. Both detectors are wire-tested independently.
|
||||
- An in-place recovery is bound to the session whose keep-alive recovered: one overtaken by a re-initialise neither
|
||||
publishes, nor degrades, nor re-arms the outage latch of the new session.
|
||||
|
||||
There is **no in-driver reconnect** any more. The SDK's `SessionReconnectHandler` (removed) retried forever in SDK
|
||||
1.5.378 (its "exhausted" path was unreachable), recreated sessions by cloning subscriptions the driver never
|
||||
re-tracked, and kept health `Healthy` throughout — so on docker-dev a stopped opc-plc left nodes Good on their last
|
||||
value indefinitely and they never recovered. A teardown against a lost session skips the wire deletes and `CloseSession`
|
||||
(bounded to 5 s otherwise). SDK warnings (keep-alive / publish / channel errors) now reach the driver's own logger,
|
||||
rate-limited to 20 per minute with a suppressed-count note; the per-driver SDK logger factory is disposed with the
|
||||
driver.
|
||||
|
||||
### Namespace assignment (retired, Gitea #648)
|
||||
|
||||
Earlier versions required a per-instance `TargetNamespaceKind` (`Equipment` +
|
||||
@@ -107,11 +154,11 @@ initialise (the AdminUI form drops them on the next save). Pinned by
|
||||
> from whatever the host retained: an inactive+acked one goes out as `Clear` + `Acknowledge` (#657; it used to be an
|
||||
> `Unspecified` the host wrote as "Activated", and a retained-Active host condition stayed Active). So the host writes no
|
||||
> duplicate row when its condition already shows that state (see `docs/AlarmTracking.md`). Condition **branch** events (non-null `BranchId`) are ignored.
|
||||
> **`ConditionRefresh`** is called after the event item is created, after every in-place reconnect, and on a
|
||||
> **`ConditionRefresh`** is called after the event item is created, after a keep-alive outage the session survived, and on a
|
||||
> `RefreshRequired` event, so a condition already active upstream is reported at subscribe; `RefreshStart`/`RefreshEnd`
|
||||
> markers are dropped. A server without it (`BadNotSupported`/`BadMethodInvalid`/…) is logged once and tolerated.
|
||||
> Last-told state survives a re-initialise (the host re-subscribes the same set over nodes it did not rebuild) and is
|
||||
> forgotten when the subscribed set changes (the host rebuilt, recreating the nodes inactive). Index keys are server-stable (`nsu=`), so a namespace reorder across a reconnect keeps
|
||||
> forgotten when the subscribed set changes (the host rebuilt, recreating the nodes inactive). Index keys are server-stable (`nsu=`), so a namespace reorder across a re-initialise keeps
|
||||
> matching. docker-dev seeds an opc-plc driver (`MAIN-opcua`); the alarm tag is
|
||||
> created only when `@OpcPlcAlarmSourceNodeId` is set in `docker-dev/seed/seed-clusters.sql`.
|
||||
|
||||
@@ -192,8 +239,10 @@ every `InitializeAsync`, including comms-loss reconnects). The discovery channel
|
||||
Session/publish events are published on the shared OTel meter as the counter
|
||||
`otopcua.driver.opcuaclient.events`, tagged `driver_instance` (the driver instance id) and `event` =
|
||||
`publish_response`, `notification` (items), `data_change`, `bad_status_value`, `publish_error`,
|
||||
`keepalive_miss`, `reconnect_attempt`, `reconnect_success`, `reconnect_exhausted`. A climbing `keepalive_miss` /
|
||||
`reconnect_attempt` rate is the signature of a flapping upstream. They are not part of `DriverHealth` or the
|
||||
`keepalive_miss`, `keepalive_recovered` (a bad keep-alive followed by a good one on the same session),
|
||||
`stale_notification_dropped` (a late replay dropped inside the post-recovery window, see above). A climbing
|
||||
`keepalive_miss` / `keepalive_recovered` rate is the signature of a flapping upstream (`reconnect_*` were removed with
|
||||
the reconnect handler, #670). They are not part of `DriverHealth` or the
|
||||
`/hosts` panel.
|
||||
|
||||
## Testing
|
||||
|
||||
@@ -0,0 +1,29 @@
|
||||
-- OtOpcUa SQL Server historian backend - schema version 2: OPC UA Part 11 modified-value history (Gitea #668).
|
||||
-- $(Schema) is substituted by SqlServerSchemaBootstrapper (identifier-validated) BEFORE the script is split on GO lines.
|
||||
-- Every statement is guarded so a re-run is a no-op.
|
||||
--
|
||||
-- One row per sample a HistoryUpdate inserted, replaced or deleted, carrying the PRIOR value (all NULL for an
|
||||
-- insert). Rows are written by the SAME statement that changes hist.Sample (OUTPUT ... INTO), so a failed update
|
||||
-- leaves no audit row and a committed one always has its row.
|
||||
-- WARNING: OUTPUT ... INTO forbids, on its target table: enabled triggers, either side of a FOREIGN KEY, and CHECK
|
||||
-- constraints. Do not add any of those to SampleModified, or every HistoryUpdate fails.
|
||||
IF OBJECT_ID(N'[$(Schema)].[SampleModified]', N'U') IS NULL
|
||||
CREATE TABLE [$(Schema)].[SampleModified] (
|
||||
TagId int NOT NULL,
|
||||
SourceTimeUtc datetime2(7) NOT NULL,
|
||||
ModifiedId bigint IDENTITY(1,1) NOT NULL,
|
||||
ModificationTimeUtc datetime2(7) NOT NULL,
|
||||
UpdateType tinyint NOT NULL,
|
||||
Principal nvarchar(200) NOT NULL,
|
||||
ApiKeyId nvarchar(200) NULL,
|
||||
ValueKind tinyint NULL,
|
||||
NumValue float NULL,
|
||||
IntValue bigint NULL,
|
||||
StrValue nvarchar(max) NULL,
|
||||
StatusCode int NULL,
|
||||
ServerTimeUtc datetime2(7) NULL,
|
||||
CONSTRAINT PK_SampleModified PRIMARY KEY CLUSTERED (TagId, SourceTimeUtc, ModifiedId))
|
||||
GO
|
||||
IF NOT EXISTS (SELECT 1 FROM [$(Schema)].[SchemaVersion] WHERE Version = 2)
|
||||
INSERT [$(Schema)].[SchemaVersion] (Version, AppliedUtc) VALUES (2, SYSUTCDATETIME())
|
||||
GO
|
||||
@@ -0,0 +1,70 @@
|
||||
# Issue batch #667–#673 — design (2026-10-11)
|
||||
|
||||
All seven open issues ≥ #667 are testable with available infra (docker-dev rig, SQL Server, opc-plc fixture on
|
||||
10.100.0.35). Base: master `ee67b637`. Recommended choices taken throughout (user-approved).
|
||||
|
||||
## Lanes
|
||||
|
||||
| Lane | Issue | Model | Wave |
|
||||
|---|---|---|---|
|
||||
| L1 | #673 FOCAS flake | sonnet | 1 |
|
||||
| L2 | #670 OpcUaClient session loss | opus | 1 |
|
||||
| L3 | #672 crash-failover capture gap | opus | 1 |
|
||||
| L4 | #669 event HistoryRead continuation points | sonnet | 1 |
|
||||
| L5 | #667 SqlServer retention/purge | sonnet | 1 |
|
||||
| L6 | #671 time-weighted Part 13 aggregates | opus | 1 |
|
||||
| L7 | #668 Part 11 modified values | opus | 2 (after L4 + L5 merge) |
|
||||
|
||||
Each lane: isolated worktree reset to master, TDD, independent opus review, merge, full suite, docker-dev live gate.
|
||||
|
||||
## #673
|
||||
Root cause: `FocasDriver.FixedTreeLoopAsync` publishes `LastStatus` before `FixedTreeCache` / `LastFixedPollUtc`;
|
||||
the test waits on `LastStatus` only, and an authored `FIXED:Status/*` read returns `BadCommunicationError` while
|
||||
`LastFixedPollUtc` is null. Fix: test waits on all three; driver writes `LastStatus` last via `Volatile.Write`.
|
||||
|
||||
## #670
|
||||
Root causes: bad keep-alive never touches `_health` nor publishes; `LastSuccessfulRead` never advances on
|
||||
notifications/keep-alives; the SDK `SessionReconnectHandler` retries forever (exhausted path unreachable) and on a
|
||||
session recreate clones subscriptions the driver's dictionaries no longer track.
|
||||
Fix (Option B): bad keep-alive ⇒ `Degraded` (stamp preserved) + one `BadNoCommunication` publish per subscribed ref
|
||||
per outage (value retained) + quality-only alarm update; good keep-alive / notifications advance
|
||||
`LastSuccessfulRead`; a self-healed short outage re-reads and republishes Good (beats the (value,status) dedup);
|
||||
sustained loss is recovered by the #527 watchdog (shutdown + init + resubscribe). Delete `SessionReconnectHandler`
|
||||
wiring, `OnReconnectComplete`, `TryRearmReconnect`, the `reconnect_exhausted` counter. Wire SDK telemetry logging
|
||||
at Warning. Correct `docs/DriverLifecycle.md` survey.
|
||||
|
||||
## #672
|
||||
Standby opens the historization capture gate when its recording Primary is cluster-UNREACHABLE (local
|
||||
`ClusterEvent.ReachabilityEvent`, recomputed from the cached redundancy snapshot in `DriverHostActor`). Alarm drain
|
||||
gate and data-plane Primary gate unchanged (falsification tests). Knob
|
||||
`ContinuousHistorization:CaptureWhilePrimaryUnreachable` (default true). No backend capability flag. Gap shrinks to
|
||||
the failure-detector window (~10–12 s). Documented: drivers without a source timestamp (Modbus) duplicate rows in
|
||||
the overlap window.
|
||||
|
||||
## #669
|
||||
Event HistoryRead issues/resumes continuation points: cursor = last event time + trailing tie count; oversized tie
|
||||
cluster over-fetch via `[T, T+resolution)` bounded by `MaxTieClusterOverfetch`; generic `SliceTieCluster`; store
|
||||
state widened to carry an event cursor; unknown CP ⇒ `BadContinuationPointInvalid`. Reverse event reads out of
|
||||
scope. Ties exact on SqlServer/QuestDB, best-effort on Gateway (documented). Adds an over-the-wire event HistoryRead
|
||||
test.
|
||||
|
||||
## #667
|
||||
`ServerHistorian:SqlServer:{RetentionDays (null=forever), PurgeInterval=1h, PurgeBatchSize=4000}`; optional
|
||||
`IHistorianBackendProvider.CreateRetention`; `HistorianRetentionService` gated on `IHistorizationRoleView`
|
||||
plus `sp_getapplock` (timeout 0); per-tag batched `DELETE TOP(n)` on the clustered key, events by time; cutoff on
|
||||
source time; metric `otopcua.historian.retention.purged{table}`. QuestDB: no change (TTL). Partition switching
|
||||
deferred/documented.
|
||||
|
||||
## #671
|
||||
New time-weighted family: TimeAverage, TimeAverage2, Interpolative, Total (now the Part 13 time-weighted integral,
|
||||
matching Gateway — client-visible, release-noted), Total2. Bounding values outside the window + `LEAD()` in-bucket
|
||||
areas in SQL; C# splits bucket-crossing segments. Sloped for numeric, stepped for Boolean. Server-default
|
||||
AggregateConfiguration (client's ignored). QuestDB mirrored after a lag/lead support spike. Hand-computed literals
|
||||
in the shared conformance suite.
|
||||
|
||||
## #668
|
||||
`hist.SampleModified` (schema v2), prior values via `OUTPUT deleted.* INTO` in the same statement;
|
||||
`HistoryUpdateContext(Principal, ApiKeyId)` threaded into updater methods; Insert operations logged (empty prior);
|
||||
retention purges never logged but #667 purges `SampleModified`; `ReadModifiedAsync` serves `IsReadModified=true`
|
||||
with continuation points, authorized by the HistoryRead gate. QuestDB: append-only `sample_modified` written before
|
||||
the change (orphan-audit-row-on-crash, documented deviation).
|
||||
@@ -884,6 +884,12 @@ HistoryUpdate rewrites recorded history, so unlike HistoryRead it is gated **wha
|
||||
- **Flag off (default):** the session's API key is re-checked on the call (`DataPlaneAuthorizer.AuthorizeKeyUsable`, Gitea #563 — an expired or disabled key loses its activation-time roles), then the session must carry the **`HistoryUpdate` role** (`OpcUaDataPlaneRoles.HistoryUpdate`). `ApiKeyInterimRoleMapper` confers it **only from a `Cluster`-scoped grant** carrying `NodePermissions.HistoryUpdate` — narrower-scoped grants confer nothing, for the same reason as `WriteOperate` (honouring a flag without honouring its scope would widen it). A key holding only `WriteOperate` is denied `BadUserAccessDenied`.
|
||||
- **Flag on:** `DataPlaneAuthorizer.Authorize(identity, OpcUaOperation.HistoryUpdate, nodeId, realm)` decides per node and **replaces** the role check (never chained) — a session with no roles succeeds on the strength of a scoped `HistoryUpdate` grant, and a `WriteOperate` grant does not help.
|
||||
- **Availability is separate from authorization.** A node accepts HistoryUpdate only if it is historized **and** the selected historian backend has an updater when the node materialises (the `HistoryWrite` AccessLevel bit). `Backend=Gateway` has none, so it answers `BadHistoryOperationUnsupported` for every node whatever the grant. Operations, result codes and unsupported cases: [`docs/Historian.md` §HistoryUpdate](Historian.md#historyupdate).
|
||||
- **Every applied change is attributed (Part 11 modified-value history, Gitea #668).** The session's principal (the API key's
|
||||
`NodeAcl` principal, or the reserved `anonymous`) and key id are passed to the backend with every HistoryUpdate and recorded in the
|
||||
modification trail next to the prior value; a modified-value HistoryRead (`IsReadModified = true`) returns it as
|
||||
`ModificationInfo.UserName`. Reading the trail is gated by the **same `HistoryRead` decision** as a raw read — there is no separate
|
||||
permission — so with `Security:Acl:Enabled` off it is as ungated as any HistoryRead, and anyone who may read a node's history can see
|
||||
who rewrote it and what it held before. See [`docs/Historian.md` §Modified-value history](Historian.md#modified-value-history-part-11).
|
||||
|
||||
### Evaluator — `PermissionTrie`
|
||||
|
||||
|
||||
@@ -56,6 +56,13 @@ public class HistoryReadCommand : CommandBase
|
||||
[CommandOption("interval", Description = "Processing interval in milliseconds for aggregates")]
|
||||
public double IntervalMs { get; init; } = 3600000;
|
||||
|
||||
/// <summary>
|
||||
/// Gets a value indicating whether to read modified-value history (Part 11 IsReadModified) instead of raw samples:
|
||||
/// the prior value each HistoryUpdate replaced or deleted, with its update type, user and modification time.
|
||||
/// </summary>
|
||||
[CommandOption("modified", Description = "Read modified-value history (what HistoryUpdate changed: prior value, update type, user, time) instead of raw samples")]
|
||||
public bool Modified { get; init; }
|
||||
|
||||
/// <inheritdoc />
|
||||
public override async ValueTask ExecuteAsync(IConsole console)
|
||||
{
|
||||
@@ -64,6 +71,8 @@ public class HistoryReadCommand : CommandBase
|
||||
throw new CommandException($"--max must be greater than 0 (was {MaxValues}).");
|
||||
if (!string.IsNullOrEmpty(Aggregate) && IntervalMs <= 0)
|
||||
throw new CommandException($"--interval must be greater than 0 (was {IntervalMs}).");
|
||||
if (Modified && !string.IsNullOrEmpty(Aggregate))
|
||||
throw new CommandException("--modified and --aggregate cannot be combined.");
|
||||
|
||||
NodeId nodeId;
|
||||
try
|
||||
@@ -119,6 +128,30 @@ public class HistoryReadCommand : CommandBase
|
||||
var ct = console.RegisterCancellationHandler();
|
||||
(service, _) = await CreateServiceAndConnectAsync(ct);
|
||||
|
||||
if (Modified)
|
||||
{
|
||||
await console.Output.WriteLineAsync(
|
||||
$"Modified history for {NodeId} ({start:yyyy-MM-dd HH:mm} -> {end:yyyy-MM-dd HH:mm})");
|
||||
var entries = await service.HistoryReadModifiedAsync(nodeId, start, end, MaxValues, ct);
|
||||
await console.Output.WriteLineAsync();
|
||||
await console.Output.WriteLineAsync(
|
||||
$"{"Timestamp",-35} {"Prior value",-15} {"Status",-10} {"Update",-8} {"User",-20} {"Modified at"}");
|
||||
foreach (var e in entries)
|
||||
{
|
||||
var dv = e.Value;
|
||||
var status = StatusCode.IsGood(dv.StatusCode) ? "Good"
|
||||
: StatusCode.IsBad(dv.StatusCode) ? "Bad"
|
||||
: "Uncertain";
|
||||
await console.Output.WriteLineAsync(
|
||||
$"{dv.SourceTimestamp.ToString("O"),-35} {dv.Value?.ToString() ?? "(none)",-15} {status,-10} " +
|
||||
$"{e.UpdateType,-8} {e.UserName ?? "",-20} {e.ModificationTime:O}");
|
||||
}
|
||||
|
||||
await console.Output.WriteLineAsync();
|
||||
await console.Output.WriteLineAsync($"{entries.Count} modifications returned.");
|
||||
return;
|
||||
}
|
||||
|
||||
IReadOnlyList<DataValue> values;
|
||||
|
||||
if (string.IsNullOrEmpty(Aggregate))
|
||||
|
||||
@@ -206,6 +206,74 @@ internal sealed class DefaultSessionAdapter : ISessionAdapter
|
||||
return allValues;
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public async Task<IReadOnlyList<Models.HistoryModifiedEntry>> HistoryReadModifiedAsync(
|
||||
NodeId nodeId, DateTime startTime, DateTime endTime, int maxValues, CancellationToken ct)
|
||||
{
|
||||
var details = new ReadRawModifiedDetails
|
||||
{
|
||||
StartTime = startTime,
|
||||
EndTime = endTime,
|
||||
NumValuesPerNode = (uint)maxValues,
|
||||
IsReadModified = true,
|
||||
ReturnBounds = false
|
||||
};
|
||||
|
||||
var nodesToRead = new HistoryReadValueIdCollection
|
||||
{
|
||||
new HistoryReadValueId { NodeId = nodeId }
|
||||
};
|
||||
|
||||
var all = new List<Models.HistoryModifiedEntry>();
|
||||
byte[]? continuationPoint = null;
|
||||
|
||||
do
|
||||
{
|
||||
if (continuationPoint != null)
|
||||
nodesToRead[0].ContinuationPoint = continuationPoint;
|
||||
|
||||
var response = await _session.HistoryReadAsync(
|
||||
null,
|
||||
new ExtensionObject(details),
|
||||
TimestampsToReturn.Both,
|
||||
false,
|
||||
nodesToRead,
|
||||
ct).ConfigureAwait(false);
|
||||
|
||||
var results = response.Results;
|
||||
if (results == null || results.Count == 0)
|
||||
break;
|
||||
|
||||
var result = results[0];
|
||||
HistoryReadStatus.ThrowIfRefused(result.StatusCode, nodeId);
|
||||
|
||||
if (result.HistoryData is ExtensionObject ext && ext.Body is HistoryModifiedData modified)
|
||||
{
|
||||
for (var i = 0; i < modified.DataValues.Count; i++)
|
||||
{
|
||||
var info = i < modified.ModificationInfos.Count ? modified.ModificationInfos[i] : null;
|
||||
all.Add(new Models.HistoryModifiedEntry(
|
||||
modified.DataValues[i],
|
||||
info?.ModificationTime ?? DateTime.MinValue,
|
||||
info?.UpdateType ?? default,
|
||||
info?.UserName));
|
||||
}
|
||||
}
|
||||
|
||||
continuationPoint = result.ContinuationPoint;
|
||||
} while (continuationPoint != null && continuationPoint.Length > 0 && all.Count < maxValues);
|
||||
|
||||
// A read stopped by the client's own cap leaves a server cursor behind; release it.
|
||||
if (continuationPoint is { Length: > 0 })
|
||||
{
|
||||
nodesToRead[0].ContinuationPoint = continuationPoint;
|
||||
await _session.HistoryReadAsync(null, new ExtensionObject(details), TimestampsToReturn.Both, true,
|
||||
nodesToRead, ct).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
return all;
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public async Task<IReadOnlyList<DataValue>> HistoryReadAggregateAsync(
|
||||
NodeId nodeId, DateTime startTime, DateTime endTime, NodeId aggregateId, double intervalMs,
|
||||
|
||||
@@ -110,6 +110,20 @@ internal interface ISessionAdapter : IDisposable
|
||||
Task<IReadOnlyList<DataValue>> HistoryReadRawAsync(NodeId nodeId, DateTime startTime, DateTime endTime,
|
||||
int maxValues, CancellationToken ct = default);
|
||||
|
||||
/// <summary>
|
||||
/// Reads modified-value history (Part 11, <c>IsReadModified = true</c>): the prior values and modification info
|
||||
/// the server recorded for every HistoryUpdate in the window, following continuation points.
|
||||
/// </summary>
|
||||
/// <param name="nodeId">The historized node.</param>
|
||||
/// <param name="startTime">Window start.</param>
|
||||
/// <param name="endTime">Window end.</param>
|
||||
/// <param name="maxValues">The maximum number of entries to return.</param>
|
||||
/// <param name="ct">The cancellation token.</param>
|
||||
/// <returns>The modification entries in server order.</returns>
|
||||
Task<IReadOnlyList<Models.HistoryModifiedEntry>> HistoryReadModifiedAsync(NodeId nodeId, DateTime startTime,
|
||||
DateTime endTime, int maxValues, CancellationToken ct = default) =>
|
||||
throw new NotSupportedException("This session adapter does not support modified-value history reads.");
|
||||
|
||||
/// <summary>
|
||||
/// Sends a HistoryUpdate request for the supplied details.
|
||||
/// </summary>
|
||||
|
||||
@@ -193,6 +193,20 @@ public interface IOpcUaClientService : IDisposable
|
||||
Task<IReadOnlyList<DataValue>> HistoryReadRawAsync(NodeId nodeId, DateTime startTime, DateTime endTime,
|
||||
int maxValues = 1000, CancellationToken ct = default);
|
||||
|
||||
/// <summary>
|
||||
/// Reads modified-value history (Part 11, <c>IsReadModified = true</c>): what each HistoryUpdate in the window
|
||||
/// replaced or deleted, with the server's modification time, update type and user name.
|
||||
/// </summary>
|
||||
/// <param name="nodeId">The historized node.</param>
|
||||
/// <param name="startTime">Window start.</param>
|
||||
/// <param name="endTime">Window end.</param>
|
||||
/// <param name="maxValues">The maximum number of entries to return.</param>
|
||||
/// <param name="ct">The cancellation token.</param>
|
||||
/// <returns>The modification entries in server order.</returns>
|
||||
Task<IReadOnlyList<HistoryModifiedEntry>> HistoryReadModifiedAsync(NodeId nodeId, DateTime startTime,
|
||||
DateTime endTime, int maxValues = 1000, CancellationToken ct = default) =>
|
||||
throw new NotSupportedException("This client service does not support modified-value history reads.");
|
||||
|
||||
/// <summary>
|
||||
/// Sends a HistoryUpdate request (insert/replace/update or delete) for the supplied details.
|
||||
/// </summary>
|
||||
|
||||
@@ -0,0 +1,17 @@
|
||||
using Opc.Ua;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Client.Shared.Models;
|
||||
|
||||
/// <summary>
|
||||
/// One entry of an OPC UA Part 11 modified-value history read (HistoryReadRaw with <c>IsReadModified = true</c>): the
|
||||
/// value as it was BEFORE the modification plus the server's <see cref="ModificationInfo"/>.
|
||||
/// </summary>
|
||||
/// <param name="Value">The prior value at its source timestamp (an insert had none: null value, Bad status).</param>
|
||||
/// <param name="ModificationTime">When the modification was applied.</param>
|
||||
/// <param name="UpdateType">The Part 11 update type (Insert / Replace / Update / Delete).</param>
|
||||
/// <param name="UserName">Who made the modification (the server's recorded principal).</param>
|
||||
public sealed record HistoryModifiedEntry(
|
||||
DataValue Value,
|
||||
DateTime ModificationTime,
|
||||
HistoryUpdateType UpdateType,
|
||||
string? UserName);
|
||||
@@ -559,6 +559,15 @@ public sealed class OpcUaClientService : IOpcUaClientService
|
||||
return await _session!.HistoryReadRawAsync(nodeId, startTime, endTime, maxValues, ct);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public async Task<IReadOnlyList<HistoryModifiedEntry>> HistoryReadModifiedAsync(
|
||||
NodeId nodeId, DateTime startTime, DateTime endTime, int maxValues = 1000, CancellationToken ct = default)
|
||||
{
|
||||
ThrowIfDisposed();
|
||||
ThrowIfNotConnected();
|
||||
return await _session!.HistoryReadModifiedAsync(nodeId, startTime, endTime, maxValues, ct);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public async Task<HistoryUpdateOutcome> HistoryUpdateAsync(HistoryUpdateDetails details, CancellationToken ct = default)
|
||||
{
|
||||
|
||||
@@ -80,11 +80,11 @@ public static class OtOpcUaTelemetry
|
||||
/// Session/publish-level events of an OPC UA Client driver instance (#276): the operator's view of a
|
||||
/// flapping upstream. Tagged <see cref="PollOverrunDriverInstanceTag"/> (<c>driver_instance</c>) +
|
||||
/// <c>event</c> = <c>publish_response|notification|data_change|bad_status_value|publish_error|
|
||||
/// keepalive_miss|reconnect_attempt|reconnect_success|reconnect_exhausted</c>.
|
||||
/// keepalive_miss|keepalive_recovered|stale_notification_dropped</c>.
|
||||
/// </summary>
|
||||
public static readonly Counter<long> OpcUaClientEvents =
|
||||
Meter.CreateCounter<long>("otopcua.driver.opcuaclient.events", unit: "{event}",
|
||||
description: "OPC UA Client driver session/publish events (event=publish_response|notification|data_change|bad_status_value|publish_error|keepalive_miss|reconnect_attempt|reconnect_success|reconnect_exhausted).");
|
||||
description: "OPC UA Client driver session/publish events (event=publish_response|notification|data_change|bad_status_value|publish_error|keepalive_miss|keepalive_recovered|stale_notification_dropped).");
|
||||
|
||||
// ---------------- VirtualTag / ScriptedAlarm engines ----------------
|
||||
|
||||
|
||||
@@ -0,0 +1,74 @@
|
||||
namespace ZB.MOM.WW.OtOpcUa.Core.Abstractions;
|
||||
|
||||
// OPC UA Part 11 modified-value history (Gitea #668): every HistoryUpdate a backend applies leaves an audit row
|
||||
// carrying the PRIOR value, who made the change and when; HistoryRead with IsReadModified=true serves them.
|
||||
|
||||
/// <summary>Who is performing a HistoryUpdate. Threaded into every <see cref="IHistorianDataUpdater"/> call.</summary>
|
||||
/// <param name="Principal">
|
||||
/// The session's authenticated principal (an API key's principal, or the reserved <c>anonymous</c>). Recorded as
|
||||
/// the modification's user name.
|
||||
/// </param>
|
||||
/// <param name="ApiKeyId">The API key id the session authenticated with, or <c>null</c> (anonymous / non-key).</param>
|
||||
public sealed record HistoryUpdateContext(string Principal, string? ApiKeyId)
|
||||
{
|
||||
/// <summary>The longest principal / key id a backend stores; longer values are truncated (never refused).</summary>
|
||||
public const int MaxIdentityLength = 200;
|
||||
|
||||
/// <summary>The principal clamped to <see cref="MaxIdentityLength"/> characters.</summary>
|
||||
public string StoredPrincipal => Clamp(Principal) ?? string.Empty;
|
||||
|
||||
/// <summary>The key id clamped to <see cref="MaxIdentityLength"/> characters.</summary>
|
||||
public string? StoredApiKeyId => Clamp(ApiKeyId);
|
||||
|
||||
private static string? Clamp(string? s) => s is { Length: > MaxIdentityLength } ? s[..MaxIdentityLength] : s;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The OPC UA Part 11 <c>HistoryUpdateType</c> of a recorded modification. The numeric values are the spec's and
|
||||
/// are persisted (SQL <c>tinyint</c>, QuestDB <c>BYTE</c>) — never renumber.
|
||||
/// </summary>
|
||||
public enum HistoryModificationType : byte
|
||||
{
|
||||
/// <summary>A sample was inserted where none existed (no prior value).</summary>
|
||||
Insert = 1,
|
||||
|
||||
/// <summary>An existing sample was replaced.</summary>
|
||||
Replace = 2,
|
||||
|
||||
/// <summary>An upsert (<c>PerformUpdateType.Update</c>): replaced a sample, or inserted one (then no prior value).</summary>
|
||||
Update = 3,
|
||||
|
||||
/// <summary>A sample was deleted (DeleteAtTime / DeleteRaw).</summary>
|
||||
Delete = 4,
|
||||
}
|
||||
|
||||
/// <summary>One recorded modification: the value as it was BEFORE the change, plus who changed it and when.</summary>
|
||||
/// <param name="Value">
|
||||
/// The prior value at its source timestamp. An insert had no prior value: <c>Value = null</c>,
|
||||
/// <c>StatusCode = BadNoData</c>.
|
||||
/// </param>
|
||||
/// <param name="ModificationTimeUtc">When the modification was applied.</param>
|
||||
/// <param name="UpdateType">The kind of modification.</param>
|
||||
/// <param name="UserName">The principal that made it.</param>
|
||||
/// <param name="ModifiedId">The backend's ordering key within one source timestamp (paging cursor).</param>
|
||||
/// <param name="SourceTimeUtc">
|
||||
/// The modified sample's source time as stored on the trail row — the paging cursor's time component. Explicit
|
||||
/// because an insert's <see cref="Value"/> has no prior value and so cannot be relied on to carry it.
|
||||
/// </param>
|
||||
public sealed record HistoryModifiedValue(
|
||||
DataValueSnapshot Value,
|
||||
DateTime ModificationTimeUtc,
|
||||
HistoryModificationType UpdateType,
|
||||
string UserName,
|
||||
long ModifiedId,
|
||||
DateTime SourceTimeUtc);
|
||||
|
||||
/// <summary>Resume position for a paged modified-value read: strictly after (or before, reversed) this key.</summary>
|
||||
/// <param name="SourceTimeUtc">The last returned modification's source time.</param>
|
||||
/// <param name="ModifiedId">The last returned modification's id.</param>
|
||||
public sealed record HistoryModifiedCursor(DateTime SourceTimeUtc, long ModifiedId);
|
||||
|
||||
/// <summary>Result of <see cref="IHistorianDataSource.ReadModifiedAsync"/>.</summary>
|
||||
/// <param name="Values">Modifications ordered by <c>(SourceTimeUtc, ModifiedId)</c> (descending for a reverse read).</param>
|
||||
/// <param name="HasMore"><c>true</c> when the cap truncated the result and more modifications remain in the window.</param>
|
||||
public sealed record HistoryModifiedReadResult(IReadOnlyList<HistoryModifiedValue> Values, bool HasMore);
|
||||
@@ -47,6 +47,12 @@ public enum HistoryAggregateType
|
||||
AnnotationCount,
|
||||
StandardDeviationPopulation,
|
||||
VariancePopulation,
|
||||
|
||||
/// <summary>Part 13 TimeAverage2: time-weighted average over the interval's non-Bad data only (#671).</summary>
|
||||
TimeAverage2,
|
||||
|
||||
/// <summary>Part 13 Total2: time-weighted integral (value·seconds) over the interval's non-Bad data only (#671).</summary>
|
||||
Total2,
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
@@ -65,6 +71,22 @@ public sealed class HistoryAggregateNotSupportedException : NotSupportedExceptio
|
||||
=> Aggregate = aggregate;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Thrown by a history backend when a read would exceed a configured size limit (e.g. the raw-row cap of the SQL
|
||||
/// backends' time-weighted aggregates, #671). The Server HistoryRead path translates it to
|
||||
/// <c>BadTooManyOperations</c> for that node only; it is a client-request limit, not a backend fault.
|
||||
/// </summary>
|
||||
public sealed class HistoryReadLimitExceededException : InvalidOperationException
|
||||
{
|
||||
/// <summary>The configured limit that was exceeded.</summary>
|
||||
public long Limit { get; }
|
||||
|
||||
/// <summary>Creates the exception.</summary>
|
||||
/// <param name="limit">The configured limit.</param>
|
||||
/// <param name="message">Operator-facing description.</param>
|
||||
public HistoryReadLimitExceededException(long limit, string message) : base(message) => Limit = limit;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// One row returned by <see cref="IHistorianDataSource.ReadEventsAsync"/> — a historical
|
||||
/// alarm/event record, not the OPC UA live-event stream. Fields match the minimum set the
|
||||
|
||||
@@ -122,4 +122,32 @@ public interface IHistorianDataSource : IDisposable
|
||||
/// Same-timestamp ties are unaffected (they are paged within the timestamp, never stepped over).
|
||||
/// </remarks>
|
||||
long TimestampResolutionTicks => 1;
|
||||
|
||||
/// <summary>
|
||||
/// Read modified-value history — OPC UA HistoryReadRaw with <c>IsReadModified = true</c> (Part 11, Gitea #668):
|
||||
/// every modification a HistoryUpdate applied to the tag inside the window, each carrying the PRIOR value.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Window: forward (<c>start < end</c>) <c>[start, end)</c> ascending by <c>(SourceTimeUtc, ModifiedId)</c>;
|
||||
/// reverse (<c>end < start</c>) <c>(end, start]</c> descending; <c>start == end</c> the modifications at
|
||||
/// exactly that time. <paramref name="resumeAfter"/> continues strictly past that key in the read's direction.
|
||||
/// The default throws <see cref="NotSupportedException"/> — a backend that keeps no modification trail
|
||||
/// (Gateway, Null) — which the server reports as <c>BadHistoryOperationUnsupported</c>.
|
||||
/// </remarks>
|
||||
/// <param name="fullReference">The historian tagname.</param>
|
||||
/// <param name="startUtc">Window start (UTC).</param>
|
||||
/// <param name="endUtc">Window end (UTC).</param>
|
||||
/// <param name="maxValues">Cap on returned modifications; <c>0</c> is unbounded.</param>
|
||||
/// <param name="resumeAfter">Resume cursor from a previous page, or <c>null</c> for the first page.</param>
|
||||
/// <param name="cancellationToken">Cancellation.</param>
|
||||
/// <returns>The modifications, with <see cref="HistoryModifiedReadResult.HasMore"/> set when truncated.</returns>
|
||||
/// <exception cref="NotSupportedException">The backend keeps no modified-value history.</exception>
|
||||
Task<HistoryModifiedReadResult> ReadModifiedAsync(
|
||||
string fullReference,
|
||||
DateTime startUtc,
|
||||
DateTime endUtc,
|
||||
uint maxValues,
|
||||
HistoryModifiedCursor? resumeAfter,
|
||||
CancellationToken cancellationToken) =>
|
||||
throw new NotSupportedException("This historian backend keeps no modified-value history.");
|
||||
}
|
||||
|
||||
@@ -17,12 +17,20 @@ public enum HistoryUpdateMode
|
||||
/// Optional historian capability behind OPC UA HistoryUpdate. A backend that cannot modify history
|
||||
/// supplies no updater (the server then reports HistoryUpdate as unsupported).
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <b>Modified-value history (Part 11, Gitea #668).</b> Every sample an implementation inserts, replaces or deletes
|
||||
/// MUST leave one modification record carrying the prior value (none for an insert), the
|
||||
/// <see cref="HistoryUpdateContext.Principal"/>, the modification time and the <see cref="HistoryModificationType"/>,
|
||||
/// readable through <see cref="IHistorianDataSource.ReadModifiedAsync"/>. A refused item (any non-Good code) leaves
|
||||
/// no record. Live recorder writes and retention purges never leave one.
|
||||
/// </remarks>
|
||||
public interface IHistorianDataUpdater
|
||||
{
|
||||
/// <summary>Inserts, replaces or upserts samples for one tag.</summary>
|
||||
/// <param name="tag">Historian tag name.</param>
|
||||
/// <param name="mode">Update mode.</param>
|
||||
/// <param name="values">Samples to apply.</param>
|
||||
/// <param name="context">Who is making the change (recorded in the modification trail).</param>
|
||||
/// <param name="ct">Cancellation token.</param>
|
||||
/// <returns>
|
||||
/// One status code per value, in order: Insert gives <c>GoodEntryInserted</c> or <c>BadEntryExists</c>;
|
||||
@@ -30,20 +38,24 @@ public interface IHistorianDataUpdater
|
||||
/// <c>GoodEntryInserted</c> or <c>GoodEntryReplaced</c>; a null timestamp gives <c>BadInvalidTimestamp</c>.
|
||||
/// </returns>
|
||||
Task<IReadOnlyList<uint>> UpdateDataAsync(string tag, HistoryUpdateMode mode,
|
||||
IReadOnlyList<HistorizationValue> values, CancellationToken ct);
|
||||
IReadOnlyList<HistorizationValue> values, HistoryUpdateContext context, CancellationToken ct);
|
||||
|
||||
/// <summary>Deletes raw samples in a time range.</summary>
|
||||
/// <param name="tag">Historian tag name.</param>
|
||||
/// <param name="startUtc">Range start (UTC).</param>
|
||||
/// <param name="endUtc">Range end (UTC).</param>
|
||||
/// <param name="context">Who is making the change (recorded in the modification trail).</param>
|
||||
/// <param name="ct">Cancellation token.</param>
|
||||
/// <returns><c>Good</c>, or <c>BadNoData</c> when the range held nothing.</returns>
|
||||
Task<uint> DeleteRawAsync(string tag, DateTime startUtc, DateTime endUtc, CancellationToken ct);
|
||||
Task<uint> DeleteRawAsync(string tag, DateTime startUtc, DateTime endUtc, HistoryUpdateContext context,
|
||||
CancellationToken ct);
|
||||
|
||||
/// <summary>Deletes the samples at exact timestamps.</summary>
|
||||
/// <param name="tag">Historian tag name.</param>
|
||||
/// <param name="timesUtc">Timestamps to delete (UTC).</param>
|
||||
/// <param name="context">Who is making the change (recorded in the modification trail).</param>
|
||||
/// <param name="ct">Cancellation token.</param>
|
||||
/// <returns>One status code per time, in order: <c>Good</c> or <c>BadNoEntryExists</c>.</returns>
|
||||
Task<IReadOnlyList<uint>> DeleteAtTimeAsync(string tag, IReadOnlyList<DateTime> timesUtc, CancellationToken ct);
|
||||
Task<IReadOnlyList<uint>> DeleteAtTimeAsync(string tag, IReadOnlyList<DateTime> timesUtc, HistoryUpdateContext context,
|
||||
CancellationToken ct);
|
||||
}
|
||||
|
||||
@@ -1232,13 +1232,16 @@ public sealed class FocasDriver : IDriver, IReadable, IWritable, ITagDiscovery,
|
||||
ProgramInfo: programInfo is not null,
|
||||
Timers: timer is not null,
|
||||
Status: statusProbe is not null);
|
||||
state.LastStatus = statusProbe;
|
||||
|
||||
state.FixedTreeCache = new FocasFixedTreeCache(sys, [.. axes], [.. spindles], caps);
|
||||
// The bootstrap IS an answer from the device, and Identity comes entirely from it —
|
||||
// so stamp liveness here as well as per tick, or a CNC whose axis poll never
|
||||
// succeeds would serve a perfectly valid sysinfo as stale.
|
||||
state.LastFixedPollUtc = DateTime.UtcNow;
|
||||
// Published LAST (release write): LastStatus is the field callers watch for "the first
|
||||
// answer is in", and an authored FIXED:Status/* read is only servable once
|
||||
// FixedTreeCache and LastFixedPollUtc are set (TryReadFixedTreeOnDevice). Publishing it
|
||||
// first let a reader see a status while the read still returned BadCommunicationError (#673).
|
||||
state.LastStatus = statusProbe;
|
||||
}
|
||||
catch (OperationCanceledException) when (ct.IsCancellationRequested) { return; }
|
||||
catch (Exception ex)
|
||||
@@ -1998,7 +2001,12 @@ public sealed class FocasDriver : IDriver, IReadable, IWritable, ITagDiscovery,
|
||||
/// </summary>
|
||||
public ConcurrentDictionary<string, byte> CoalesceRejected { get; } = new(StringComparer.Ordinal);
|
||||
/// <summary>Gets or sets the last <c>cnc_statinfo</c> snapshot (#257); null until the first answer.</summary>
|
||||
public FocasStatus? LastStatus { get; set; }
|
||||
public FocasStatus? LastStatus
|
||||
{
|
||||
get => Volatile.Read(ref _lastStatus);
|
||||
set => Volatile.Write(ref _lastStatus, value);
|
||||
}
|
||||
private FocasStatus? _lastStatus;
|
||||
/// <summary>Gets or sets the last program information snapshot.</summary>
|
||||
public FocasProgramInfo? LastProgramInfo { get; set; }
|
||||
/// <summary>Gets or sets the cached first-axis dynamic snapshot — feeds Program/Number, /MainNumber, /Sequence.</summary>
|
||||
|
||||
@@ -105,6 +105,42 @@ internal static class QuestDbAggregateQueries
|
||||
"SELECT st.*, r.num_value, r.ts " +
|
||||
"FROM st LEFT JOIN (SELECT * FROM r WHERE rn = 1) r ON r.b = st.b ORDER BY st.b";
|
||||
|
||||
/// <summary>
|
||||
/// Time-weighted family (#671), bounds statement — <b>not</b> pushed down (see <see cref="TimeWeightedWindow"/>).
|
||||
/// Four <c>LIMIT 1</c> seeks <c>UNION ALL</c>-ed, each tagged with <c>k</c>: 0 = latest usable sample before the
|
||||
/// start, 1 = latest sample of any quality before the start, 2 = first sample of any quality at or after the end,
|
||||
/// 3 = first usable one at or after the end. Parameters: <c>$1</c> tag, <c>$2</c>/<c>$3</c> the forward window
|
||||
/// start / end. Columns: <c>k</c>, then <c>ts, kind, num_value, int_value, str_value, status, server_ts</c>.
|
||||
/// </summary>
|
||||
/// <param name="sample">Prefixed sample table name.</param>
|
||||
/// <returns>The SQL text.</returns>
|
||||
public static string TimeWeightedBounds(string sample)
|
||||
{
|
||||
var usable = $" AND status < {NotBadLimit} AND kind <> {EmptyKind}";
|
||||
string Seek(int k, string cmp, string filter, string order) =>
|
||||
$"SELECT * FROM (SELECT {k.ToString(CultureInfo.InvariantCulture)} AS k, {SampleColumns} FROM {sample} " +
|
||||
$"WHERE tag = $1 AND ts {cmp}{filter} AND NOT deleted ORDER BY ts {order} LIMIT 1)";
|
||||
return Seek(0, "< $2", usable, "DESC") + " UNION ALL " + Seek(1, "< $2", "", "DESC") + " UNION ALL " +
|
||||
Seek(2, ">= $3", "", "ASC") + " UNION ALL " + Seek(3, ">= $3", usable, "ASC");
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Time-weighted family (#671), window statement: the live samples in <c>[$2, $3)</c> for tag <c>$1</c>, ascending,
|
||||
/// at most <c>$4</c> (the raw-row cap + 1).
|
||||
/// Columns: <c>ts, kind, num_value, int_value, str_value, status, server_ts</c>. The caller feeds the before-bounds,
|
||||
/// these rows and the after-bounds, in that order, into <see cref="TimeWeightedAggregator"/> — the same evaluator
|
||||
/// the SQL Server backend uses, so the two agree exactly. Trade-off: the window's rows travel to the server rather
|
||||
/// than one row per bucket; Part 13's segment interpolation and bucket-edge splitting are then written once
|
||||
/// instead of twice in two SQL dialects that must stay bit-identical.
|
||||
/// </summary>
|
||||
/// <param name="sample">Prefixed sample table name.</param>
|
||||
/// <returns>The SQL text.</returns>
|
||||
public static string TimeWeightedWindow(string sample) =>
|
||||
$"SELECT {SampleColumns} FROM {sample} WHERE tag = $1 AND ts >= $2 AND ts < $3 AND NOT deleted ORDER BY ts ASC LIMIT $4";
|
||||
|
||||
/// <summary>The row columns the time-weighted statements return (after <c>k</c> for the bounds).</summary>
|
||||
private const string SampleColumns = "ts, kind, num_value, int_value, str_value, status, server_ts";
|
||||
|
||||
/// <summary>The bucket-index expression (exposed for tests).</summary>
|
||||
internal const string BucketExpression = "(cast(ts AS LONG) * 10 - $1) / $2";
|
||||
|
||||
|
||||
@@ -83,6 +83,8 @@ internal sealed class QuestDbConnection : IAsyncDisposable, IDisposable
|
||||
public string EventTable => T(QuestDbTables.EventBase);
|
||||
/// <summary>The prefixed tag table.</summary>
|
||||
public string TagTable => T(QuestDbTables.TagBase);
|
||||
/// <summary>The prefixed Part 11 modification-trail table (#668).</summary>
|
||||
public string SampleModifiedTable => T(QuestDbTables.SampleModifiedBase);
|
||||
|
||||
/// <summary>Opens a pooled connection (no bootstrap; callers run the bootstrapper first).</summary>
|
||||
/// <param name="ct">Cancellation.</param>
|
||||
|
||||
+157
-3
@@ -64,6 +64,7 @@ public sealed class QuestDbHistorianDataSource : IHistorianDataSource
|
||||
private readonly ILogger _log;
|
||||
private readonly TimeProvider _time;
|
||||
private readonly HistorianHealthCounters _health = new();
|
||||
private readonly int _maxTimeWeightedRawRows;
|
||||
|
||||
/// <summary>Creates the data source over the provider's shared connection, bootstrapper and tag catalog.</summary>
|
||||
/// <param name="conn">Connection helper.</param>
|
||||
@@ -71,10 +72,12 @@ public sealed class QuestDbHistorianDataSource : IHistorianDataSource
|
||||
/// <param name="catalog">Tag catalog (reads use only its read-only <see cref="QuestDbTagCatalog.ExistsAsync"/>).</param>
|
||||
/// <param name="log">Optional logger.</param>
|
||||
/// <param name="timeProvider">Clock for "now" (AtTime beyond the last sample, server timestamps); system by default.</param>
|
||||
/// <param name="maxTimeWeightedRawRows">Raw in-window row cap of one time-weighted read (<= 0 = unbounded).</param>
|
||||
internal QuestDbHistorianDataSource(
|
||||
QuestDbConnection conn, QuestDbSchemaBootstrapper bootstrap, QuestDbTagCatalog catalog, ILogger? log = null,
|
||||
TimeProvider? timeProvider = null)
|
||||
TimeProvider? timeProvider = null, int maxTimeWeightedRawRows = TimeWeightedAggregator.DefaultMaxWindowRows)
|
||||
{
|
||||
_maxTimeWeightedRawRows = maxTimeWeightedRawRows;
|
||||
_conn = conn ?? throw new ArgumentNullException(nameof(conn));
|
||||
_bootstrap = bootstrap ?? throw new ArgumentNullException(nameof(bootstrap));
|
||||
_catalog = catalog ?? throw new ArgumentNullException(nameof(catalog));
|
||||
@@ -137,6 +140,101 @@ public sealed class QuestDbHistorianDataSource : IHistorianDataSource
|
||||
}
|
||||
}
|
||||
|
||||
// ------------------------------------------------------------------------------------------ Modified (Part 11, #668)
|
||||
|
||||
/// <inheritdoc />
|
||||
/// <remarks>
|
||||
/// Reads <c>{prefix}sample_modified</c>, the append-only trail <see cref="QuestDbHistorianDataUpdater"/> writes
|
||||
/// BEFORE each change. Windows follow <see cref="ReadRawAsync"/> (bounds rounded to µs); rows are ordered by
|
||||
/// <c>(ts, modified_id)</c>; <c>maxValues + 1</c> rows are fetched to report
|
||||
/// <see cref="HistoryModifiedReadResult.HasMore"/> exactly. A row whose <c>status</c> is NULL had no prior value
|
||||
/// (an insert) and is served <c>null</c> / <c>BadNoData</c>. Because the trail row lands first, a crash between it and
|
||||
/// the sample write leaves an orphan row describing a change that never applied (documented).
|
||||
/// </remarks>
|
||||
public async Task<HistoryModifiedReadResult> ReadModifiedAsync(
|
||||
string fullReference, DateTime startUtc, DateTime endUtc, uint maxValues, HistoryModifiedCursor? resumeAfter,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(fullReference);
|
||||
var bounded = maxValues > 0 && maxValues < int.MaxValue;
|
||||
try
|
||||
{
|
||||
var values = new List<HistoryModifiedValue>();
|
||||
var hasMore = false;
|
||||
if (await ResolveAsync(fullReference, cancellationToken).ConfigureAwait(false))
|
||||
{
|
||||
var start = QuestDbConnection.Round(startUtc);
|
||||
var end = QuestDbConnection.Round(endUtc);
|
||||
var reverse = start > end;
|
||||
string where;
|
||||
if (start == end) where = "ts = $2 AND ts = $3";
|
||||
else if (!reverse) where = "ts >= $2 AND ts < $3";
|
||||
else where = "ts > $2 AND ts <= $3";
|
||||
var next = 4;
|
||||
if (resumeAfter is not null)
|
||||
{
|
||||
where += reverse
|
||||
? " AND (ts < $4 OR (ts = $4 AND modified_id < $5))"
|
||||
: " AND (ts > $4 OR (ts = $4 AND modified_id > $5))";
|
||||
next = 6;
|
||||
}
|
||||
var order = reverse ? "DESC" : "ASC";
|
||||
var (lo, hi) = start <= end ? (start, end) : (end, start);
|
||||
|
||||
await using var conn = await _conn.OpenAsync(cancellationToken).ConfigureAwait(false);
|
||||
await using var cmd = _conn.CreateCommand(conn,
|
||||
$"SELECT {SampleColumns}, modified_id, modification_ts, update_type, principal FROM {_conn.SampleModifiedTable} " +
|
||||
$"WHERE tag = $1 AND {where} ORDER BY ts {order}, modified_id {order}" + (bounded ? $" LIMIT ${next}" : ""));
|
||||
QuestDbConnection.BindText(cmd, fullReference);
|
||||
QuestDbConnection.BindTimestamp(cmd, lo);
|
||||
QuestDbConnection.BindTimestamp(cmd, hi);
|
||||
if (resumeAfter is not null)
|
||||
{
|
||||
QuestDbConnection.BindTimestamp(cmd, resumeAfter.SourceTimeUtc);
|
||||
QuestDbConnection.BindLong(cmd, resumeAfter.ModifiedId);
|
||||
}
|
||||
if (bounded)
|
||||
cmd.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Integer, Value = (int)maxValues + 1 });
|
||||
|
||||
await using var reader = await cmd.ExecuteReaderAsync(cancellationToken).ConfigureAwait(false);
|
||||
while (await reader.ReadAsync(cancellationToken).ConfigureAwait(false))
|
||||
{
|
||||
if (bounded && values.Count == maxValues)
|
||||
{
|
||||
hasMore = true;
|
||||
break;
|
||||
}
|
||||
|
||||
var sourceTime = QuestDbConnection.ReadUtc(reader, 0);
|
||||
var modifiedAt = QuestDbConnection.ReadUtc(reader, 8);
|
||||
DataValueSnapshot prior;
|
||||
if (reader.IsDBNull(5))
|
||||
{
|
||||
prior = new DataValueSnapshot(null, HistorianStatusCodes.BadNoData, sourceTime, modifiedAt);
|
||||
}
|
||||
else
|
||||
{
|
||||
var s = ReadStored(reader, 0);
|
||||
prior = new DataValueSnapshot(s.Value.ToObject(), s.Status, s.TimeUtc, ReadServerTime(reader, 6, modifiedAt));
|
||||
}
|
||||
|
||||
values.Add(new HistoryModifiedValue(prior, modifiedAt,
|
||||
(HistoryModificationType)QuestDbConnection.ReadKind(reader, 9),
|
||||
reader.IsDBNull(10) ? string.Empty : reader.GetString(10),
|
||||
reader.GetInt64(7), sourceTime));
|
||||
}
|
||||
}
|
||||
|
||||
RecordSuccess();
|
||||
return new HistoryModifiedReadResult(values, hasMore);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
RecordFailure(ex);
|
||||
throw;
|
||||
}
|
||||
}
|
||||
|
||||
// ------------------------------------------------------------------------------------------ AtTime
|
||||
|
||||
/// <inheritdoc />
|
||||
@@ -332,9 +430,10 @@ public sealed class QuestDbHistorianDataSource : IHistorianDataSource
|
||||
/// the <see cref="BucketRow"/> statistics of each non-empty bucket (bucket index
|
||||
/// <c>(cast(ts AS LONG)*10 − StartUnixTicks) / IntervalTicks</c>, never <c>SAMPLE BY</c>);
|
||||
/// <see cref="AggregateEvaluator.Evaluate"/> produces each value and <see cref="BucketPlan.Assemble"/> fills the
|
||||
/// empty buckets with <c>BadNoData</c>.
|
||||
/// empty buckets with <c>BadNoData</c>. The time-weighted family (#671) instead streams the window plus four bounding
|
||||
/// rows into <see cref="TimeWeightedAggregator"/>, exactly as the SQL Server backend does.
|
||||
/// </remarks>
|
||||
/// <exception cref="HistoryAggregateNotSupportedException">The aggregate is not one of the thirteen served.</exception>
|
||||
/// <exception cref="HistoryAggregateNotSupportedException">The aggregate is not one of the seventeen served.</exception>
|
||||
/// <exception cref="ArgumentOutOfRangeException">The window holds more than <see cref="BucketPlan.MaxBuckets"/> intervals.</exception>
|
||||
public async Task<HistoryReadResult> ReadProcessedAsync(
|
||||
string fullReference, DateTime startUtc, DateTime endUtc, TimeSpan interval,
|
||||
@@ -348,6 +447,15 @@ public sealed class QuestDbHistorianDataSource : IHistorianDataSource
|
||||
{
|
||||
var now = _time.GetUtcNow().UtcDateTime;
|
||||
var evaluated = new Dictionary<long, DataValueSnapshot>();
|
||||
if (family == AggregateFamily.TimeWeighted)
|
||||
{
|
||||
var tw = new TimeWeightedAggregator(aggregate, plan);
|
||||
if (plan.BucketCount > 0 && await ResolveAsync(fullReference, cancellationToken).ConfigureAwait(false))
|
||||
await StreamTimeWeightedAsync(fullReference, plan, tw, cancellationToken).ConfigureAwait(false);
|
||||
var twResult = tw.Finish(now);
|
||||
RecordSuccess();
|
||||
return new HistoryReadResult(twResult, ContinuationPoint: null);
|
||||
}
|
||||
if (plan.BucketCount > 0 && await ResolveAsync(fullReference, cancellationToken).ConfigureAwait(false))
|
||||
{
|
||||
var sample = _conn.SampleTable;
|
||||
@@ -377,6 +485,11 @@ public sealed class QuestDbHistorianDataSource : IHistorianDataSource
|
||||
RecordSuccess();
|
||||
return new HistoryReadResult(result, ContinuationPoint: null);
|
||||
}
|
||||
catch (HistoryReadLimitExceededException)
|
||||
{
|
||||
// A client-request limit (the raw-row cap), not a backend fault: not counted as a failure.
|
||||
throw;
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
RecordFailure(ex);
|
||||
@@ -384,6 +497,47 @@ public sealed class QuestDbHistorianDataSource : IHistorianDataSource
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Streams the time-weighted bounds (<see cref="QuestDbAggregateQueries.TimeWeightedBounds"/>) and window
|
||||
/// (<see cref="QuestDbAggregateQueries.TimeWeightedWindow"/>) into <paramref name="tw"/> in time order: the two
|
||||
/// before-bounds, the window, the two after-bounds (duplicates are dropped by the aggregator).
|
||||
/// </summary>
|
||||
private async Task StreamTimeWeightedAsync(string tag, BucketPlan plan, TimeWeightedAggregator tw, CancellationToken ct)
|
||||
{
|
||||
var bounds = new StoredSample?[4];
|
||||
await using var conn = await _conn.OpenAsync(ct).ConfigureAwait(false);
|
||||
await using (var cmd = _conn.CreateCommand(conn, QuestDbAggregateQueries.TimeWeightedBounds(_conn.SampleTable)))
|
||||
{
|
||||
QuestDbConnection.BindText(cmd, tag);
|
||||
QuestDbConnection.BindTimestamp(cmd, plan.StartUtc);
|
||||
QuestDbConnection.BindTimestamp(cmd, plan.EndUtc);
|
||||
await using var reader = await cmd.ExecuteReaderAsync(ct).ConfigureAwait(false);
|
||||
while (await reader.ReadAsync(ct).ConfigureAwait(false))
|
||||
bounds[Convert.ToInt32(reader.GetValue(0), CultureInfo.InvariantCulture)] = ReadStored(reader, 1);
|
||||
}
|
||||
|
||||
if (bounds[0] is { } usableBefore) tw.Add(usableBefore);
|
||||
if (bounds[1] is { } anyBefore) tw.Add(anyBefore);
|
||||
await using (var cmd = _conn.CreateCommand(conn, QuestDbAggregateQueries.TimeWeightedWindow(_conn.SampleTable)))
|
||||
{
|
||||
QuestDbConnection.BindText(cmd, tag);
|
||||
QuestDbConnection.BindTimestamp(cmd, plan.StartUtc);
|
||||
QuestDbConnection.BindTimestamp(cmd, plan.EndUtc);
|
||||
cmd.Parameters.Add(new NpgsqlParameter
|
||||
{ NpgsqlDbType = NpgsqlDbType.Integer, Value = TimeWeightedAggregator.WindowFetchLimit(_maxTimeWeightedRawRows) });
|
||||
await using var reader = await cmd.ExecuteReaderAsync(ct).ConfigureAwait(false);
|
||||
long windowRows = 0;
|
||||
while (await reader.ReadAsync(ct).ConfigureAwait(false))
|
||||
{
|
||||
// LIMIT $4 = cap + 1: one row past the cap fails the node without reading on.
|
||||
TimeWeightedAggregator.CheckWindowRows(++windowRows, _maxTimeWeightedRawRows, tag);
|
||||
tw.Add(ReadStored(reader, 0));
|
||||
}
|
||||
}
|
||||
if (bounds[2] is { } anyAfter) tw.Add(anyAfter);
|
||||
if (bounds[3] is { } usableAfter) tw.Add(usableAfter);
|
||||
}
|
||||
|
||||
private static BucketRow ReadBucket(NpgsqlDataReader reader, AggregateFamily family)
|
||||
{
|
||||
static double? D(NpgsqlDataReader r, int i) => r.IsDBNull(i) ? null : r.GetDouble(i);
|
||||
|
||||
+184
-17
@@ -25,6 +25,13 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb;
|
||||
/// 1970 (read bounds are not clamped).</para>
|
||||
/// <para><b>Read-your-writes:</b> after writing, <see cref="QuestDbWalVisibility"/> waits for the rows to become
|
||||
/// visible, bounded by <c>VisibilityTimeout</c>; a timeout only logs a Warning, never changes a code.</para>
|
||||
/// <para><b>Modified-value history (Part 11, #668).</b> Every sample this updater inserts, replaces or tombstones first
|
||||
/// gets a row in <c>{prefix}sample_modified</c> carrying its PRIOR live value (the existence read already fetches it;
|
||||
/// none for an insert — a tombstone is "absent", so an insert over one records no prior value), the caller's
|
||||
/// <see cref="HistoryUpdateContext"/>, the <see cref="HistoryModificationType"/> and the modification time. QuestDB has
|
||||
/// no multi-table transaction, so the trail row is written <b>BEFORE</b> the change: a crash (or a failed sample INSERT)
|
||||
/// between the two leaves an orphan trail row describing a change that never landed — never a change without its
|
||||
/// trail row. This is the documented deviation from SQL Server, whose trail commits atomically with the change.</para>
|
||||
/// </summary>
|
||||
internal sealed class QuestDbHistorianDataUpdater : IHistorianDataUpdater
|
||||
{
|
||||
@@ -35,6 +42,7 @@ internal sealed class QuestDbHistorianDataUpdater : IHistorianDataUpdater
|
||||
private const int LockStripes = 64;
|
||||
private readonly SemaphoreSlim[] _locks = Enumerable.Range(0, LockStripes).Select(_ => new SemaphoreSlim(1, 1)).ToArray();
|
||||
private long _liveRowsRead;
|
||||
private static long _lastModifiedId;
|
||||
private readonly QuestDbConnection _conn;
|
||||
private readonly QuestDbTagCatalog _tags;
|
||||
private readonly QuestDbWalVisibility _wal;
|
||||
@@ -71,10 +79,11 @@ internal sealed class QuestDbHistorianDataUpdater : IHistorianDataUpdater
|
||||
|
||||
/// <inheritdoc/>
|
||||
public async Task<IReadOnlyList<uint>> UpdateDataAsync(string tag, HistoryUpdateMode mode,
|
||||
IReadOnlyList<HistorizationValue> values, CancellationToken ct)
|
||||
IReadOnlyList<HistorizationValue> values, HistoryUpdateContext context, CancellationToken ct)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(tag);
|
||||
ArgumentNullException.ThrowIfNull(values);
|
||||
ArgumentNullException.ThrowIfNull(context);
|
||||
|
||||
// RC1: a valid-timestamp item before 1970 can never be stored; refuse it and plan the rest.
|
||||
var final = new uint[values.Count];
|
||||
@@ -100,7 +109,7 @@ internal sealed class QuestDbHistorianDataUpdater : IHistorianDataUpdater
|
||||
? await _tags.ExistsAsync(tag, ct).ConfigureAwait(false)
|
||||
: await EnsureAsync(tag, plan.Winners[0].Row.Value.Kind, ct).ConfigureAwait(false);
|
||||
var outcome = known
|
||||
? await ApplyAsync(tag, mode, plan.Winners.Select(w => w.Row).ToList(), ct).ConfigureAwait(false)
|
||||
? await ApplyAsync(tag, mode, plan.Winners.Select(w => w.Row).ToList(), context, ct).ConfigureAwait(false)
|
||||
: new Dictionary<long, UpsertOutcome>();
|
||||
planned = HistoryUpdatePlanner.Complete(plan, mode, outcome);
|
||||
}
|
||||
@@ -109,9 +118,11 @@ internal sealed class QuestDbHistorianDataUpdater : IHistorianDataUpdater
|
||||
}
|
||||
|
||||
/// <inheritdoc/>
|
||||
public async Task<uint> DeleteRawAsync(string tag, DateTime startUtc, DateTime endUtc, CancellationToken ct)
|
||||
public async Task<uint> DeleteRawAsync(string tag, DateTime startUtc, DateTime endUtc, HistoryUpdateContext context,
|
||||
CancellationToken ct)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(tag);
|
||||
ArgumentNullException.ThrowIfNull(context);
|
||||
if (!HistorianTagNames.IsValid(tag)) return HistorianStatusCodes.BadInvalidArgument;
|
||||
if (!await _tags.ExistsAsync(tag, ct).ConfigureAwait(false))
|
||||
return HistoryUpdatePlanner.DeleteRawResult(tagKnown: false, 0);
|
||||
@@ -123,18 +134,29 @@ internal sealed class QuestDbHistorianDataUpdater : IHistorianDataUpdater
|
||||
while (cursor < end)
|
||||
{
|
||||
var page = new List<DateTime>(BatchSize);
|
||||
var priors = new List<LiveRow>(BatchSize);
|
||||
await using (var conn = await _conn.OpenAsync(ct).ConfigureAwait(false))
|
||||
await using (var cmd = _conn.CreateCommand(conn,
|
||||
$"SELECT ts FROM {_conn.SampleTable} WHERE tag = $1 AND ts >= $2 AND ts < $3 AND NOT deleted ORDER BY ts LIMIT {BatchSize}"))
|
||||
$"SELECT {LiveColumns} FROM {_conn.SampleTable} WHERE tag = $1 AND ts >= $2 AND ts < $3 AND NOT deleted ORDER BY ts LIMIT {BatchSize}"))
|
||||
{
|
||||
QuestDbConnection.BindText(cmd, tag);
|
||||
QuestDbConnection.BindTimestamp(cmd, cursor);
|
||||
QuestDbConnection.BindTimestamp(cmd, end);
|
||||
await using var rd = await cmd.ExecuteReaderAsync(ct).ConfigureAwait(false);
|
||||
while (await rd.ReadAsync(ct).ConfigureAwait(false)) page.Add(QuestDbConnection.ReadUtc(rd, 0));
|
||||
while (await rd.ReadAsync(ct).ConfigureAwait(false))
|
||||
{
|
||||
var row = ReadLiveRow(rd);
|
||||
page.Add(row.TimeUtc);
|
||||
priors.Add(row);
|
||||
}
|
||||
}
|
||||
if (page.Count == 0) break;
|
||||
await WriteTombstonesAsync(tag, page, ct).ConfigureAwait(false);
|
||||
// Trail BEFORE the change (no multi-table transaction): a crash here leaves an orphan trail row, never a
|
||||
// tombstone without its trail row.
|
||||
await WriteTrailAsync(tag, HistoryModificationType.Delete,
|
||||
priors.Select(r => (r.TimeUtc, (LiveRow?)r)).ToList(), context, ct).ConfigureAwait(false);
|
||||
await AfterTrailAsync(tag, page[0], page[^1], page.Count,
|
||||
() => WriteTombstonesAsync(tag, page, ct)).ConfigureAwait(false);
|
||||
// The wait between pages is required: without it the next page could re-read rows just tombstoned.
|
||||
await _wal.WaitAsync(_conn.SampleTable, ct).ConfigureAwait(false);
|
||||
total += page.Count;
|
||||
@@ -145,10 +167,12 @@ internal sealed class QuestDbHistorianDataUpdater : IHistorianDataUpdater
|
||||
}
|
||||
|
||||
/// <inheritdoc/>
|
||||
public async Task<IReadOnlyList<uint>> DeleteAtTimeAsync(string tag, IReadOnlyList<DateTime> timesUtc, CancellationToken ct)
|
||||
public async Task<IReadOnlyList<uint>> DeleteAtTimeAsync(string tag, IReadOnlyList<DateTime> timesUtc,
|
||||
HistoryUpdateContext context, CancellationToken ct)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(tag);
|
||||
ArgumentNullException.ThrowIfNull(timesUtc);
|
||||
ArgumentNullException.ThrowIfNull(context);
|
||||
if (!HistorianTagNames.IsValid(tag))
|
||||
{
|
||||
var refused = new uint[timesUtc.Count];
|
||||
@@ -168,9 +192,16 @@ internal sealed class QuestDbHistorianDataUpdater : IHistorianDataUpdater
|
||||
|
||||
using var _ = await LockAsync(tag, ct).ConfigureAwait(false);
|
||||
var live = await ReadLiveAsync(tag, candidates.Select(t => new DateTime(t, DateTimeKind.Utc)).ToList(), ct).ConfigureAwait(false);
|
||||
var toDelete = candidates.Where(live.Contains).OrderBy(x => x).ToList();
|
||||
var toDelete = candidates.Where(live.ContainsKey).OrderBy(x => x).ToList();
|
||||
foreach (var chunk in toDelete.Chunk(BatchSize))
|
||||
await WriteTombstonesAsync(tag, chunk.Select(t => new DateTime(t, DateTimeKind.Utc)).ToList(), ct).ConfigureAwait(false);
|
||||
{
|
||||
// Trail BEFORE the change (see the class remarks).
|
||||
await WriteTrailAsync(tag, HistoryModificationType.Delete,
|
||||
chunk.Select(t => (live[t].TimeUtc, (LiveRow?)live[t])).ToList(), context, ct).ConfigureAwait(false);
|
||||
var chunkTimes = chunk.Select(t => new DateTime(t, DateTimeKind.Utc)).ToList();
|
||||
await AfterTrailAsync(tag, chunkTimes[0], chunkTimes[^1], chunkTimes.Count,
|
||||
() => WriteTombstonesAsync(tag, chunkTimes, ct)).ConfigureAwait(false);
|
||||
}
|
||||
if (toDelete.Count > 0) await _wal.WaitAsync(_conn.SampleTable, ct).ConfigureAwait(false);
|
||||
deleted.UnionWith(toDelete);
|
||||
return HistoryUpdatePlanner.DeleteAtTimeResults(rounded, deleted, tagKnown: true);
|
||||
@@ -183,48 +214,184 @@ internal sealed class QuestDbHistorianDataUpdater : IHistorianDataUpdater
|
||||
}
|
||||
|
||||
private async Task<Dictionary<long, UpsertOutcome>> ApplyAsync(string tag, HistoryUpdateMode mode,
|
||||
List<HistorianSampleRow> rows, CancellationToken ct)
|
||||
List<HistorianSampleRow> rows, HistoryUpdateContext context, CancellationToken ct)
|
||||
{
|
||||
// Key = the planner's tick (what Complete looks up); the stored time is Round(of the same row).
|
||||
var live = await ReadLiveAsync(tag, rows.Select(r => QuestDbConnection.Round(r.TimeUtc)).ToList(), ct).ConfigureAwait(false);
|
||||
var outcome = new Dictionary<long, UpsertOutcome>();
|
||||
var toWrite = new List<HistorianSampleRow>();
|
||||
var trail = new List<(DateTime, LiveRow?)>();
|
||||
foreach (var r in rows)
|
||||
{
|
||||
var present = live.Contains(QuestDbConnection.Round(r.TimeUtc).Ticks);
|
||||
var stored = QuestDbConnection.Round(r.TimeUtc);
|
||||
var present = live.TryGetValue(stored.Ticks, out var prior);
|
||||
if (mode == HistoryUpdateMode.Insert && present) continue;
|
||||
if (mode == HistoryUpdateMode.Replace && !present) continue;
|
||||
outcome[r.TimeUtc.Ticks] = present ? UpsertOutcome.Replaced : UpsertOutcome.Inserted;
|
||||
toWrite.Add(r);
|
||||
trail.Add((stored, present ? prior : null));
|
||||
}
|
||||
// Trail BEFORE the change (see the class remarks).
|
||||
await WriteTrailAsync(tag, ToModificationType(mode), trail, context, ct).ConfigureAwait(false);
|
||||
foreach (var chunk in toWrite.Chunk(BatchSize))
|
||||
await WriteRowsAsync(tag, chunk, ct).ConfigureAwait(false);
|
||||
await AfterTrailAsync(tag, chunk[0].TimeUtc, chunk[^1].TimeUtc, chunk.Length,
|
||||
() => WriteRowsAsync(tag, chunk, ct)).ConfigureAwait(false);
|
||||
if (toWrite.Count > 0) await _wal.WaitAsync(_conn.SampleTable, ct).ConfigureAwait(false);
|
||||
return outcome;
|
||||
}
|
||||
|
||||
/// <summary>Ticks of the live (present, not deleted) rows near the given times: one range query per cluster (I-2).</summary>
|
||||
private async Task<HashSet<long>> ReadLiveAsync(string tag, IReadOnlyList<DateTime> times, CancellationToken ct)
|
||||
/// <summary>The live (present, not deleted) rows near the given times, keyed by stored ticks: one range query per cluster (I-2).</summary>
|
||||
private async Task<Dictionary<long, LiveRow>> ReadLiveAsync(string tag, IReadOnlyList<DateTime> times, CancellationToken ct)
|
||||
{
|
||||
var live = new HashSet<long>();
|
||||
var live = new Dictionary<long, LiveRow>();
|
||||
await using var conn = await _conn.OpenAsync(ct).ConfigureAwait(false);
|
||||
foreach (var (lo, hi) in QuestDbTimeClusters.Cluster(times, QuestDbTimeClusters.ExistenceGap))
|
||||
{
|
||||
await using var cmd = _conn.CreateCommand(conn,
|
||||
$"SELECT ts FROM {_conn.SampleTable} WHERE tag = $1 AND ts >= $2 AND ts <= $3 AND NOT deleted");
|
||||
$"SELECT {LiveColumns} FROM {_conn.SampleTable} WHERE tag = $1 AND ts >= $2 AND ts <= $3 AND NOT deleted");
|
||||
QuestDbConnection.BindText(cmd, tag);
|
||||
QuestDbConnection.BindTimestamp(cmd, lo);
|
||||
QuestDbConnection.BindTimestamp(cmd, hi);
|
||||
await using var rd = await cmd.ExecuteReaderAsync(ct).ConfigureAwait(false);
|
||||
while (await rd.ReadAsync(ct).ConfigureAwait(false))
|
||||
{
|
||||
live.Add(QuestDbConnection.ReadUtc(rd, 0).Ticks);
|
||||
var row = ReadLiveRow(rd);
|
||||
live[row.TimeUtc.Ticks] = row;
|
||||
Interlocked.Increment(ref _liveRowsRead);
|
||||
}
|
||||
}
|
||||
return live;
|
||||
}
|
||||
|
||||
/// <summary>The columns of a live sample the trail needs (prior value + status + server time).</summary>
|
||||
private const string LiveColumns = "ts, kind, num_value, int_value, str_value, status, server_ts";
|
||||
|
||||
/// <summary>A live sample as it was before a modification.</summary>
|
||||
private readonly record struct LiveRow(DateTime TimeUtc, HistorianValue Value, uint Status, DateTime? ServerTimeUtc);
|
||||
|
||||
private static LiveRow ReadLiveRow(NpgsqlDataReader rd)
|
||||
{
|
||||
var value = QuestDbHistorianDataSource.ToValue(
|
||||
QuestDbConnection.ReadKind(rd, 1),
|
||||
rd.IsDBNull(2) ? null : rd.GetDouble(2),
|
||||
rd.IsDBNull(3) ? null : rd.GetInt64(3),
|
||||
rd.IsDBNull(4) ? null : rd.GetString(4));
|
||||
var status = rd.IsDBNull(5) ? HistorianStatusCodes.BadNoData : SampleColumnMapper.FromLongStatus(rd.GetInt64(5));
|
||||
return new LiveRow(QuestDbConnection.ReadUtc(rd, 0), value, status,
|
||||
rd.IsDBNull(6) ? null : QuestDbConnection.ReadUtc(rd, 6));
|
||||
}
|
||||
|
||||
/// <summary>The Part 11 modification type recorded for an update mode.</summary>
|
||||
private static HistoryModificationType ToModificationType(HistoryUpdateMode mode) => mode switch
|
||||
{
|
||||
HistoryUpdateMode.Insert => HistoryModificationType.Insert,
|
||||
HistoryUpdateMode.Replace => HistoryModificationType.Replace,
|
||||
_ => HistoryModificationType.Update,
|
||||
};
|
||||
|
||||
/// <summary>
|
||||
/// A process-wide, strictly increasing modification id (≈ UTC ticks, bumped past the last issued value). It orders a
|
||||
/// tag's trail within one source time; ids from two nodes interleave by wall clock, which is the modification order.
|
||||
/// </summary>
|
||||
internal static long NextModifiedId()
|
||||
{
|
||||
while (true)
|
||||
{
|
||||
var last = Interlocked.Read(ref _lastModifiedId);
|
||||
var next = Math.Max(DateTime.UtcNow.Ticks, last + 1);
|
||||
if (Interlocked.CompareExchange(ref _lastModifiedId, next, last) == last) return next;
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Appends one modification-trail row per entry (<c>prior = null</c> ⇒ an insert: kind Empty, NULL value and
|
||||
/// status), then waits for the trail's WAL visibility so a modified read right after the update sees it.
|
||||
/// </summary>
|
||||
private async Task WriteTrailAsync(string tag, HistoryModificationType type, IReadOnlyList<(DateTime TimeUtc, LiveRow? Prior)> entries,
|
||||
HistoryUpdateContext context, CancellationToken ct)
|
||||
{
|
||||
if (entries.Count == 0) return;
|
||||
var now = DateTime.UtcNow;
|
||||
foreach (var chunk in entries.Chunk(BatchSize))
|
||||
{
|
||||
await using var conn = await _conn.OpenAsync(ct).ConfigureAwait(false);
|
||||
await using var cmd = _conn.CreateCommand(conn, "");
|
||||
var sb = new System.Text.StringBuilder(
|
||||
$"INSERT INTO {_conn.SampleModifiedTable} (ts, tag, modified_id, modification_ts, update_type, principal, " +
|
||||
"api_key_id, kind, num_value, int_value, str_value, status, server_ts) VALUES ");
|
||||
for (var i = 0; i < chunk.Length; i++)
|
||||
{
|
||||
var (time, prior) = chunk[i];
|
||||
QuestDbConnection.BindTimestamp(cmd, time, clampToEpoch: true);
|
||||
QuestDbConnection.BindText(cmd, tag);
|
||||
QuestDbConnection.BindLong(cmd, NextModifiedId());
|
||||
QuestDbConnection.BindTimestamp(cmd, now);
|
||||
QuestDbConnection.BindByte(cmd, (byte)type);
|
||||
QuestDbConnection.BindText(cmd, context.StoredPrincipal);
|
||||
QuestDbConnection.BindText(cmd, context.StoredApiKeyId);
|
||||
if (prior is { } p)
|
||||
{
|
||||
var c = SampleColumnMapper.ToColumns(p.Value);
|
||||
QuestDbConnection.BindByte(cmd, c.ValueKind);
|
||||
Typed(cmd, NpgsqlDbType.Double, c.NumValue);
|
||||
Typed(cmd, NpgsqlDbType.Bigint, c.IntValue);
|
||||
Typed(cmd, NpgsqlDbType.Varchar, c.StrValue);
|
||||
QuestDbConnection.BindLong(cmd, SampleColumnMapper.ToLongStatus(p.Status));
|
||||
Typed(cmd, NpgsqlDbType.TimestampTz, p.ServerTimeUtc);
|
||||
}
|
||||
else
|
||||
{
|
||||
// No prior value: QuestDB BYTE has no NULL, so kind is Empty (0) and the NULL status marks "none".
|
||||
QuestDbConnection.BindByte(cmd, (byte)HistorianValueKind.Empty);
|
||||
Typed(cmd, NpgsqlDbType.Double, null);
|
||||
Typed(cmd, NpgsqlDbType.Bigint, null);
|
||||
Typed(cmd, NpgsqlDbType.Varchar, null);
|
||||
Typed(cmd, NpgsqlDbType.Bigint, null);
|
||||
Typed(cmd, NpgsqlDbType.TimestampTz, null);
|
||||
}
|
||||
|
||||
var b = i * 13;
|
||||
if (i > 0) sb.Append(',');
|
||||
sb.Append('(');
|
||||
for (var q = 1; q <= 13; q++) sb.Append(q > 1 ? "," : "").Append('$').Append(b + q);
|
||||
sb.Append(')');
|
||||
}
|
||||
cmd.CommandText = sb.ToString();
|
||||
try
|
||||
{
|
||||
await cmd.ExecuteNonQueryAsync(ct).ConfigureAwait(false);
|
||||
}
|
||||
catch (Exception ex) when (ex is not OperationCanceledException)
|
||||
{
|
||||
_log.LogWarning(ex, "QuestDB modification-trail INSERT of {Count} rows into {Table} failed; the update was not applied.",
|
||||
chunk.Length, _conn.SampleModifiedTable);
|
||||
throw;
|
||||
}
|
||||
}
|
||||
await _wal.WaitAsync(_conn.SampleModifiedTable, ct).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Runs the sample write that follows an already-written trail batch. QuestDB has no multi-table transaction, so a
|
||||
/// failure here leaves an ORPHAN trail row (a recorded modification that never landed); it is logged with the tag
|
||||
/// and time range so an operator can reconcile it, then rethrown unchanged.
|
||||
/// </summary>
|
||||
private async Task AfterTrailAsync(string tag, DateTime from, DateTime to, int count, Func<Task> write)
|
||||
{
|
||||
try
|
||||
{
|
||||
await write().ConfigureAwait(false);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_log.LogWarning(ex,
|
||||
"QuestDB sample write failed AFTER its {Table} trail rows were written; {Count} orphan audit row(s) for tag {Tag} " +
|
||||
"in [{From:O}, {To:O}] describe a modification that did not land.",
|
||||
_conn.SampleModifiedTable, count, tag, from, to);
|
||||
throw;
|
||||
}
|
||||
}
|
||||
|
||||
private Task WriteRowsAsync(string tag, IReadOnlyList<HistorianSampleRow> rows, CancellationToken ct) =>
|
||||
InsertAsync(rows.Count, ct, (cmd, i) =>
|
||||
{
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common;
|
||||
using System.Collections.Frozen;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
@@ -33,11 +34,16 @@ public sealed class QuestDbHistorianProvider : IHistorianBackendProvider, IAsync
|
||||
|
||||
/// <summary>Creates the provider over the <c>ServerHistorian:QuestDb</c> options (validated elsewhere).</summary>
|
||||
/// <param name="options">The <c>ServerHistorian:QuestDb</c> options.</param>
|
||||
public QuestDbHistorianProvider(QuestDbHistorianOptions options)
|
||||
/// <param name="maxTimeWeightedRawRows">The shared <c>ServerHistorian:MaxTimeWeightedRawRows</c> cap (<= 0 = unbounded).</param>
|
||||
public QuestDbHistorianProvider(QuestDbHistorianOptions options,
|
||||
int maxTimeWeightedRawRows = TimeWeightedAggregator.DefaultMaxWindowRows)
|
||||
{
|
||||
_options = options ?? throw new ArgumentNullException(nameof(options));
|
||||
_maxTimeWeightedRawRows = maxTimeWeightedRawRows;
|
||||
}
|
||||
|
||||
private readonly int _maxTimeWeightedRawRows;
|
||||
|
||||
/// <inheritdoc />
|
||||
public string Name => "QuestDb";
|
||||
|
||||
@@ -69,7 +75,7 @@ public sealed class QuestDbHistorianProvider : IHistorianBackendProvider, IAsync
|
||||
{
|
||||
var s = Get(sp);
|
||||
return new QuestDbHistorianDataSource(s.Conn, s.Bootstrap, s.Catalog,
|
||||
Loggers(sp).CreateLogger<QuestDbHistorianDataSource>());
|
||||
Loggers(sp).CreateLogger<QuestDbHistorianDataSource>(), maxTimeWeightedRawRows: _maxTimeWeightedRawRows);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
|
||||
@@ -6,7 +6,7 @@ using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common;
|
||||
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb;
|
||||
|
||||
/// <summary>
|
||||
/// Lazily creates the three tables once per instance (<c>CREATE TABLE IF NOT EXISTS</c>), then reconciles the TTL
|
||||
/// Lazily creates the four tables once per instance (<c>CREATE TABLE IF NOT EXISTS</c>), then reconciles the TTL
|
||||
/// of the sample and event tables with <see cref="QuestDbHistorianOptions.RetentionDays"/>.
|
||||
/// <list type="bullet">
|
||||
/// <item><b>Concurrency:</b> two nodes starting together are safe. If a CREATE still fails with a server error
|
||||
|
||||
@@ -15,6 +15,8 @@ internal static class QuestDbTables
|
||||
public const string EventBase = "event";
|
||||
/// <summary>Base name of the tag catalog table.</summary>
|
||||
public const string TagBase = "tag";
|
||||
/// <summary>Base name of the Part 11 modification-trail table (#668).</summary>
|
||||
public const string SampleModifiedBase = "sample_modified";
|
||||
|
||||
/// <summary>The fixed designated timestamp of every <c>tag</c> row, which makes the dedup key an upsert by name.</summary>
|
||||
public static readonly DateTime TagEpoch = DateTime.UnixEpoch;
|
||||
@@ -25,12 +27,17 @@ internal static class QuestDbTables
|
||||
public static string Event(string prefix) => Name(prefix, EventBase);
|
||||
/// <summary>The prefixed tag table name.</summary>
|
||||
public static string Tag(string prefix) => Name(prefix, TagBase);
|
||||
/// <summary>The prefixed modification-trail table name.</summary>
|
||||
public static string SampleModified(string prefix) => Name(prefix, SampleModifiedBase);
|
||||
|
||||
/// <summary>All three table names in creation order.</summary>
|
||||
public static string[] All(string prefix) => [Sample(prefix), Event(prefix), Tag(prefix)];
|
||||
/// <summary>All four table names in creation order.</summary>
|
||||
public static string[] All(string prefix) => [Sample(prefix), Event(prefix), Tag(prefix), SampleModified(prefix)];
|
||||
|
||||
/// <summary>The tables that carry a TTL (sample and event; the tag catalog never expires).</summary>
|
||||
public static string[] Expiring(string prefix) => [Sample(prefix), Event(prefix)];
|
||||
/// <summary>
|
||||
/// The tables that carry a TTL (sample, event and the modification trail, which ages out with the samples it
|
||||
/// describes; the tag catalog never expires).
|
||||
/// </summary>
|
||||
public static string[] Expiring(string prefix) => [Sample(prefix), Event(prefix), SampleModified(prefix)];
|
||||
|
||||
/// <summary><c>CREATE TABLE IF NOT EXISTS</c> for the sample table.</summary>
|
||||
public static string CreateSample(string prefix, int? retentionDays) =>
|
||||
@@ -50,9 +57,22 @@ internal static class QuestDbTables
|
||||
$"CREATE TABLE IF NOT EXISTS {Tag(prefix)} (ts TIMESTAMP, name SYMBOL, data_type BYTE, eng_unit VARCHAR, " +
|
||||
"description VARCHAR, updated_utc TIMESTAMP) TIMESTAMP(ts) PARTITION BY YEAR WAL DEDUP UPSERT KEYS(ts, name)";
|
||||
|
||||
/// <summary>The three CREATE statements in creation order.</summary>
|
||||
/// <summary>
|
||||
/// <c>CREATE TABLE IF NOT EXISTS</c> for the Part 11 modification trail (#668): one row per sample a HistoryUpdate
|
||||
/// inserted, replaced or deleted, carrying the PRIOR value (NULL kind/value/status for an insert). <c>ts</c> is the
|
||||
/// sample's source time (so the TTL ages it out with the samples); <c>modified_id</c> orders the trail within one
|
||||
/// source time. Append-only: no DEDUP, never updated or deleted.
|
||||
/// </summary>
|
||||
public static string CreateSampleModified(string prefix, int? retentionDays) =>
|
||||
$"CREATE TABLE IF NOT EXISTS {SampleModified(prefix)} (ts TIMESTAMP, tag SYMBOL, modified_id LONG, " +
|
||||
"modification_ts TIMESTAMP, update_type BYTE, principal VARCHAR, api_key_id VARCHAR, kind BYTE, num_value DOUBLE, " +
|
||||
"int_value LONG, str_value VARCHAR, status LONG, server_ts TIMESTAMP) " +
|
||||
$"TIMESTAMP(ts) PARTITION BY DAY{Ttl(retentionDays)} WAL";
|
||||
|
||||
/// <summary>The four CREATE statements in creation order (parallel to <see cref="All"/>).</summary>
|
||||
public static string[] CreateAll(string prefix, int? retentionDays) =>
|
||||
[CreateSample(prefix, retentionDays), CreateEvent(prefix, retentionDays), CreateTag(prefix)];
|
||||
[CreateSample(prefix, retentionDays), CreateEvent(prefix, retentionDays), CreateTag(prefix),
|
||||
CreateSampleModified(prefix, retentionDays)];
|
||||
|
||||
/// <summary><c>ALTER TABLE t SET TTL n DAYS</c>; <c>null</c> removes the TTL (<c>SET TTL 0 DAYS</c>).</summary>
|
||||
public static string AlterTtl(string table, int? retentionDays) =>
|
||||
|
||||
@@ -17,7 +17,8 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common;
|
||||
/// <para>
|
||||
/// Values: Count is <see cref="int"/>; Start/End keep the sample's type; every other aggregate is
|
||||
/// <see cref="double"/> computed over the numeric value (Boolean = 0/1). These are the simple per-sample
|
||||
/// definitions, not Part 13's interval-boundary (interpolated bound / time-weighted) semantics.
|
||||
/// definitions; the Part 13 time-weighted aggregates (TimeAverage, TimeAverage2, Total, Total2, Interpolative)
|
||||
/// are evaluated by <see cref="TimeWeightedAggregator"/>, not here.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public static class AggregateEvaluator
|
||||
@@ -28,7 +29,8 @@ public static class AggregateEvaluator
|
||||
/// <param name="bucketStart">The bucket's start time (the result timestamp).</param>
|
||||
/// <param name="now">Server timestamp.</param>
|
||||
/// <returns>The calculated value.</returns>
|
||||
/// <exception cref="HistoryAggregateNotSupportedException">The aggregate is not one of the thirteen served.</exception>
|
||||
/// <exception cref="HistoryAggregateNotSupportedException">The aggregate is not one of the per-sample (Simple /
|
||||
/// StartEnd / ActualTime family) aggregates.</exception>
|
||||
public static DataValueSnapshot Evaluate(HistoryAggregateType aggregate, BucketRow r, DateTime bucketStart, DateTime now)
|
||||
{
|
||||
DataValueSnapshot Calc(object? value, uint status, DateTime? t = null) =>
|
||||
@@ -53,7 +55,6 @@ public static class AggregateEvaluator
|
||||
case HistoryAggregateType.Average: return Calc(r.Avg, status);
|
||||
case HistoryAggregateType.Minimum: return Calc(r.Min, status);
|
||||
case HistoryAggregateType.Maximum: return Calc(r.Max, status);
|
||||
case HistoryAggregateType.Total: return Calc(r.Sum, status);
|
||||
case HistoryAggregateType.Range: return Calc(r.Max - r.Min, status);
|
||||
case HistoryAggregateType.Start: return Calc(r.First.ToObject(), status);
|
||||
case HistoryAggregateType.End: return Calc(r.Last.ToObject(), status);
|
||||
|
||||
@@ -5,7 +5,7 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common;
|
||||
/// <summary>The query shape that serves an aggregate. One query per family.</summary>
|
||||
public enum AggregateFamily
|
||||
{
|
||||
/// <summary>Average / Minimum / Maximum / Total / Count / Range / PercentGood / PercentBad: one GROUP BY.</summary>
|
||||
/// <summary>Average / Minimum / Maximum / Count / Range / PercentGood / PercentBad: one GROUP BY.</summary>
|
||||
Simple,
|
||||
|
||||
/// <summary>Start / End / Delta: first + last usable sample per bucket.</summary>
|
||||
@@ -13,6 +13,13 @@ public enum AggregateFamily
|
||||
|
||||
/// <summary>MinimumActualTime / MaximumActualTime: extreme usable sample per bucket, ties ⇒ earliest.</summary>
|
||||
ActualTime,
|
||||
|
||||
/// <summary>
|
||||
/// TimeAverage / TimeAverage2 / Total / Total2 / Interpolative: Part 13 time-weighted, with bounding values.
|
||||
/// The backend streams the window's samples plus the bounding rows and <see cref="TimeWeightedAggregator"/>
|
||||
/// evaluates them (identically for every backend).
|
||||
/// </summary>
|
||||
TimeWeighted,
|
||||
}
|
||||
|
||||
/// <summary>Which aggregates the SQL historian backends serve, and by which family.</summary>
|
||||
@@ -24,10 +31,12 @@ public static class AggregateFamilies
|
||||
public static AggregateFamily? FamilyOf(HistoryAggregateType aggregate) => aggregate switch
|
||||
{
|
||||
HistoryAggregateType.Average or HistoryAggregateType.Minimum or HistoryAggregateType.Maximum
|
||||
or HistoryAggregateType.Total or HistoryAggregateType.Count or HistoryAggregateType.Range
|
||||
or HistoryAggregateType.Count or HistoryAggregateType.Range
|
||||
or HistoryAggregateType.PercentGood or HistoryAggregateType.PercentBad => AggregateFamily.Simple,
|
||||
HistoryAggregateType.Start or HistoryAggregateType.End or HistoryAggregateType.Delta => AggregateFamily.StartEnd,
|
||||
HistoryAggregateType.MinimumActualTime or HistoryAggregateType.MaximumActualTime => AggregateFamily.ActualTime,
|
||||
HistoryAggregateType.TimeAverage or HistoryAggregateType.TimeAverage2 or HistoryAggregateType.Total
|
||||
or HistoryAggregateType.Total2 or HistoryAggregateType.Interpolative => AggregateFamily.TimeWeighted,
|
||||
_ => null,
|
||||
};
|
||||
}
|
||||
|
||||
@@ -12,6 +12,12 @@ public static class HistorianInfoBits
|
||||
/// <summary>The value was interpolated.</summary>
|
||||
public const uint HistorianInterpolated = 0x2u;
|
||||
|
||||
/// <summary>
|
||||
/// Partial (Part 4 §7.39 / <c>Opc.Ua.AggregateBits.Partial</c>): the calculated value covers an interval shorter
|
||||
/// than the processing interval (the truncated last interval of a processed read).
|
||||
/// </summary>
|
||||
public const uint HistorianPartial = 0x4u;
|
||||
|
||||
/// <summary>
|
||||
/// Marks <paramref name="status"/> as a Calculated data value. Any historian bits the status already carries
|
||||
/// (e.g. a stored Interpolated sample) are cleared first, so the result is never both.
|
||||
|
||||
@@ -10,6 +10,9 @@ public static class HistorianStatusCodes
|
||||
public const uint GoodEntryInserted = 0x00A20000u;
|
||||
public const uint GoodEntryReplaced = 0x00A30000u;
|
||||
public const uint BadNoData = 0x809B0000u;
|
||||
|
||||
/// <summary>Generic Bad: a time-weighted interval holding data but no non-Bad time (the SDK aggregate rule).</summary>
|
||||
public const uint Bad = 0x80000000u;
|
||||
public const uint BadEntryExists = 0x809F0000u;
|
||||
public const uint BadNoEntryExists = 0x80A00000u;
|
||||
public const uint BadInvalidTimestamp = 0x80230000u;
|
||||
|
||||
@@ -0,0 +1,430 @@
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common;
|
||||
|
||||
/// <summary>
|
||||
/// The OPC UA Part 13 time-weighted aggregates (<see cref="AggregateFamily.TimeWeighted"/>): <c>TimeAverage</c>,
|
||||
/// <c>TimeAverage2</c>, <c>Total</c>, <c>Total2</c> and <c>Interpolative</c>, evaluated over a time-ordered stream of
|
||||
/// stored samples. Both SQL backends fetch the same rows and feed them here, so the two produce identical results
|
||||
/// by construction.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// <b>Input.</b> Samples in strictly ascending time: the bounding rows before the window (the latest
|
||||
/// <i>usable</i> sample before <c>start</c>, then the latest sample of any quality before <c>start</c> — they differ
|
||||
/// when an outage straddles the start), every in-window sample, then the bounding rows at/after the window end
|
||||
/// (the first sample of any quality at or after <c>end</c>, then the first <i>usable</i> one). <see cref="Add"/>
|
||||
/// ignores a sample at or before the last one, so a bound fetched twice is harmless. Memory is O(buckets), not O(samples).
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>The signal (server defaults: TreatUncertainAsBad = false, PercentDataBad = PercentDataGood = 100,
|
||||
/// UseSlopedExtrapolation = false; a client <c>AggregateConfiguration</c> is not honoured).</b> Between a usable
|
||||
/// sample <c>p</c> and the next sample <c>s</c>:
|
||||
/// <list type="bullet">
|
||||
/// <item><c>s</c> usable, both Double/Int64 ⇒ <b>sloped</b> (linear <c>p → s</c>); quality = worse of the two.</item>
|
||||
/// <item><c>s</c> usable, either Boolean ⇒ <b>stepped</b> (<c>p</c> held); quality = <c>p</c>'s.</item>
|
||||
/// <item><c>s</c> Bad ⇒ <c>p</c> held up to the outage (never interpolated into it, as AtTime does); Uncertain.</item>
|
||||
/// <item>no later sample at all ⇒ <c>p</c> held to the window end (stepped extrapolation); Uncertain.</item>
|
||||
/// </list>
|
||||
/// From a Bad sample (incl. an <c>Empty</c> outage marker) to the next sample, and before the first sample, the
|
||||
/// signal is Bad (no data). This is the <i>simple bounding</i> signal.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// The <i>interpolated bounding</i> signal (Part 13 "interpolated bounding values") skips Bad samples: it joins
|
||||
/// consecutive <b>usable</b> samples the same way (sloped / stepped) straight across any outage, and holds the
|
||||
/// last usable value to the window end. It is undefined before the first usable sample.
|
||||
/// </para>
|
||||
/// <list type="table">
|
||||
/// <item><term>TimeAverage</term><description>∫ interpolated-bounding signal ÷ its defined duration in the interval.</description></item>
|
||||
/// <item><term>Total</term><description><c>TimeAverage × interval seconds</c> (value·s) — Part 13 §5.4.3.8.</description></item>
|
||||
/// <item><term>TimeAverage2</term><description>∫ simple-bounding signal over its non-Bad parts ÷ non-Bad duration (Bad intervals excluded).</description></item>
|
||||
/// <item><term>Total2</term><description>∫ simple-bounding signal over its non-Bad parts, in value·s (= TimeAverage2 × non-Bad seconds).</description></item>
|
||||
/// <item><term>Interpolative</term><description>The interpolated-bounding signal's value at each interval start —
|
||||
/// for a reverse read, at <c>End − i·interval</c> (Part 13 anchors a reverse read at its later time), emitted
|
||||
/// latest first.</description></item>
|
||||
/// </list>
|
||||
/// <para>
|
||||
/// <b>Status (time-weighted four)</b> — the SDK <c>AggregateCalculator.GetTimeBasedStatusCode</c> rule with
|
||||
/// TreatUncertainAsBad = false, so Uncertain time (Uncertain samples, a value held up to an outage, stepped
|
||||
/// extrapolation past the last sample) counts as Good time: every instant of the interval non-Bad ⇒
|
||||
/// <c>Good</c>; no non-Bad time but some Bad (outage) time ⇒ <c>Bad</c>; no data at all ⇒ <c>BadNoData</c>;
|
||||
/// otherwise (PercentDataGood = PercentDataBad = 100) <c>UncertainDataSubNormal</c>. Time before the first sample
|
||||
/// counts as Bad (no data) time. A usable String/DateTime sample contributing to the interval ⇒
|
||||
/// <c>BadAggregateInvalidInputs</c>. Every status carries the Calculated bit, plus the Partial bit on a non-Bad
|
||||
/// result of a truncated last interval; the timestamp is the interval start; the value is a
|
||||
/// <see cref="double"/> (Boolean = 0/1, Int64 as double).
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Status (Interpolative).</b> A usable sample exactly at the point ⇒ its value with its own status
|
||||
/// (raw, no historian bit). Otherwise no usable sample at or before ⇒ <c>BadNoData</c>; else the interpolated
|
||||
/// value with the Interpolated bit and the status of the simple-bounding signal at that instant: <c>Good</c> on
|
||||
/// a Good segment, <c>UncertainDataSubNormal</c> on an Uncertain one, inside an outage (interpolated across it)
|
||||
/// or past the last sample (extrapolated).
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public sealed class TimeWeightedAggregator
|
||||
{
|
||||
private const double TicksPerSecond = TimeSpan.TicksPerSecond;
|
||||
|
||||
/// <summary>
|
||||
/// Default cap on the raw in-window rows one time-weighted read may stream (<c>ServerHistorian:MaxTimeWeightedRawRows</c>).
|
||||
/// </summary>
|
||||
public const int DefaultMaxWindowRows = 1_000_000;
|
||||
|
||||
/// <summary>
|
||||
/// The <c>TOP</c>/<c>LIMIT</c> a backend puts on its window query for a configured cap: <c>cap + 1</c> (so one
|
||||
/// row past the cap proves it was exceeded without reading on), or <see cref="int.MaxValue"/> when the cap is
|
||||
/// <= 0 (unbounded).
|
||||
/// </summary>
|
||||
/// <param name="maxWindowRows">The configured cap.</param>
|
||||
/// <returns>The row limit to put on the window query.</returns>
|
||||
public static int WindowFetchLimit(int maxWindowRows) =>
|
||||
maxWindowRows <= 0 || maxWindowRows == int.MaxValue ? int.MaxValue : maxWindowRows + 1;
|
||||
|
||||
/// <summary>Throws when <paramref name="rowsRead"/> has passed a positive <paramref name="maxWindowRows"/>.</summary>
|
||||
/// <param name="rowsRead">Window rows read so far.</param>
|
||||
/// <param name="maxWindowRows">The configured cap (<= 0 = unbounded).</param>
|
||||
/// <param name="tag">The tag, for the message.</param>
|
||||
/// <exception cref="HistoryReadLimitExceededException">The cap is exceeded.</exception>
|
||||
public static void CheckWindowRows(long rowsRead, int maxWindowRows, string tag)
|
||||
{
|
||||
if (maxWindowRows > 0 && rowsRead > maxWindowRows)
|
||||
throw new HistoryReadLimitExceededException(maxWindowRows,
|
||||
$"The time-weighted aggregate read of '{tag}' covers more than {maxWindowRows} raw samples " +
|
||||
"(ServerHistorian:MaxTimeWeightedRawRows); narrow the window or use a pushed-down aggregate such as Average.");
|
||||
}
|
||||
|
||||
private readonly HistoryAggregateType _aggregate;
|
||||
private readonly BucketPlan _plan;
|
||||
private readonly long _start;
|
||||
private readonly long _end;
|
||||
|
||||
// Simple-bounding (raw) signal, per bucket: ∫ over non-Bad time, non-Bad ticks, ticks inside an outage.
|
||||
private readonly double[] _rawArea;
|
||||
private readonly long[] _usableTicks;
|
||||
private readonly long[] _badTicks;
|
||||
// Interpolated-bounding (bridged) signal, per bucket.
|
||||
private readonly double[] _bridgedArea;
|
||||
private readonly long[] _bridgedTicks;
|
||||
private readonly bool[] _invalid;
|
||||
// Interpolative, per point (index = ascending time; see PointTicks).
|
||||
private readonly DataValueSnapshot?[] _points;
|
||||
|
||||
private StoredSample? _prevAny;
|
||||
private StoredSample? _prevUsable;
|
||||
private long _nextPoint; // next point (ascending time) not yet classified
|
||||
private readonly List<PendingPoint> _pending = []; // points awaiting the next usable sample
|
||||
private bool _finished;
|
||||
|
||||
/// <summary>
|
||||
/// What the simple-bounding signal is at a pending bucket start: no sample yet; on a segment from a usable sample
|
||||
/// whose end is not known yet; or degraded (inside an outage, or held up to one).
|
||||
/// </summary>
|
||||
private enum PointQuality { NoPrior, OnSegment, Degraded }
|
||||
|
||||
private readonly record struct PendingPoint(long Bucket, long Ticks, PointQuality Quality);
|
||||
|
||||
/// <summary>Starts an evaluation of <paramref name="aggregate"/> over <paramref name="plan"/>.</summary>
|
||||
/// <param name="aggregate">One of the <see cref="AggregateFamily.TimeWeighted"/> aggregates.</param>
|
||||
/// <param name="plan">The bucket plan (forward window).</param>
|
||||
/// <exception cref="HistoryAggregateNotSupportedException">The aggregate is not time-weighted.</exception>
|
||||
public TimeWeightedAggregator(HistoryAggregateType aggregate, BucketPlan plan)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(plan);
|
||||
if (AggregateFamilies.FamilyOf(aggregate) != AggregateFamily.TimeWeighted)
|
||||
throw new HistoryAggregateNotSupportedException(aggregate);
|
||||
_aggregate = aggregate;
|
||||
_plan = plan;
|
||||
_start = plan.StartUtc.Ticks;
|
||||
_end = plan.EndUtc.Ticks;
|
||||
var n = (int)plan.BucketCount;
|
||||
_rawArea = new double[n];
|
||||
_usableTicks = new long[n];
|
||||
_badTicks = new long[n];
|
||||
_bridgedArea = new double[n];
|
||||
_bridgedTicks = new long[n];
|
||||
_invalid = new bool[n];
|
||||
_points = new DataValueSnapshot?[n];
|
||||
}
|
||||
|
||||
/// <summary>Evaluates <paramref name="aggregate"/> over an already time-ordered sample sequence.</summary>
|
||||
/// <param name="aggregate">One of the <see cref="AggregateFamily.TimeWeighted"/> aggregates.</param>
|
||||
/// <param name="plan">The bucket plan.</param>
|
||||
/// <param name="samples">Bounds + window samples, ascending by time.</param>
|
||||
/// <param name="now">Server timestamp for the results.</param>
|
||||
/// <returns>One value per bucket, in the plan's order (descending for a reverse window).</returns>
|
||||
public static IReadOnlyList<DataValueSnapshot> Evaluate(
|
||||
HistoryAggregateType aggregate, BucketPlan plan, IEnumerable<StoredSample> samples, DateTime now)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(samples);
|
||||
var acc = new TimeWeightedAggregator(aggregate, plan);
|
||||
foreach (var s in samples) acc.Add(s);
|
||||
return acc.Finish(now);
|
||||
}
|
||||
|
||||
/// <summary>Feeds the next sample (ascending time; a sample at or before the previous one is ignored).</summary>
|
||||
/// <param name="s">The sample.</param>
|
||||
public void Add(StoredSample s)
|
||||
{
|
||||
if (_finished) throw new InvalidOperationException("The aggregation is already finished.");
|
||||
var t = s.TimeUtc.Ticks;
|
||||
if (_prevAny is { } last && t <= last.TimeUtc.Ticks) return;
|
||||
|
||||
// Interpolative: classify every bucket start before this sample by the simple-bounding signal there.
|
||||
ClassifyPointsBefore(t);
|
||||
|
||||
// Simple-bounding segment [prevAny, s).
|
||||
if (_prevAny is { IsUsable: true } p)
|
||||
{
|
||||
// Toward a usable sample: sloped or stepped; toward a Bad one: held up to the outage.
|
||||
var sloped = s.IsUsable && IsSloped(p, s);
|
||||
AddRaw(p, t, sloped ? s.Value.Number : p.Value.Number);
|
||||
}
|
||||
else if (_prevAny is { } bad)
|
||||
{
|
||||
AddBad(bad, t);
|
||||
}
|
||||
|
||||
if (!s.IsUsable) DegradePending(); // the segment the pending points sit on runs into an outage
|
||||
|
||||
if (s.IsUsable)
|
||||
{
|
||||
if (_prevUsable is { } pu)
|
||||
{
|
||||
var sloped = IsSloped(pu, s);
|
||||
AddBridged(pu, t, sloped ? s.Value.Number : pu.Value.Number);
|
||||
}
|
||||
ResolvePending(_prevUsable, s);
|
||||
_prevUsable = s;
|
||||
}
|
||||
|
||||
_prevAny = s;
|
||||
}
|
||||
|
||||
/// <summary>Completes the evaluation (extrapolating past the last sample) and assembles the per-bucket results.</summary>
|
||||
/// <param name="now">Server timestamp for the results.</param>
|
||||
/// <returns>One value per bucket, in the plan's order (descending for a reverse window).</returns>
|
||||
public IReadOnlyList<DataValueSnapshot> Finish(DateTime now)
|
||||
{
|
||||
if (!_finished)
|
||||
{
|
||||
_finished = true;
|
||||
ClassifyPointsBefore(long.MaxValue);
|
||||
if (_prevAny is { IsUsable: true } p && p.TimeUtc.Ticks < _end)
|
||||
AddRaw(p, _end, p.Value.Number); // stepped extrapolation
|
||||
else if (_prevAny is { } bad && bad.TimeUtc.Ticks < _end)
|
||||
AddBad(bad, _end); // the outage runs to the window end
|
||||
if (_prevUsable is { } pu && pu.TimeUtc.Ticks < _end)
|
||||
AddBridged(pu, _end, pu.Value.Number);
|
||||
ResolvePending(_prevUsable, next: null);
|
||||
}
|
||||
|
||||
var byBucket = new Dictionary<long, DataValueSnapshot>((int)_plan.BucketCount);
|
||||
for (long i = 0; i < _plan.BucketCount; i++)
|
||||
byBucket[i] = Result(i, now);
|
||||
return _plan.Assemble(byBucket, t => AggregateEvaluator.NoData(t, now));
|
||||
}
|
||||
|
||||
// ------------------------------------------------------------------------------------------ Results
|
||||
|
||||
private DataValueSnapshot Result(long i, DateTime now)
|
||||
{
|
||||
if (_aggregate == HistoryAggregateType.Interpolative)
|
||||
return _points[i] is { } point
|
||||
? point with { ServerTimestampUtc = now }
|
||||
: new DataValueSnapshot(null, HistorianStatusCodes.BadNoData, PointTime(i), now);
|
||||
|
||||
var bucketStart = _plan.BucketStart(i);
|
||||
var duration = BucketEnd(i) - BucketStartTicks(i);
|
||||
var partial = duration < _plan.IntervalTicks; // only the truncated last interval
|
||||
DataValueSnapshot Calc(object? value, uint status)
|
||||
{
|
||||
var code = HistorianInfoBits.WithCalculated(status);
|
||||
if (partial && !StatusBits.IsBad(status)) code |= HistorianInfoBits.HistorianPartial;
|
||||
return new DataValueSnapshot(value, code, bucketStart, now);
|
||||
}
|
||||
|
||||
if (_invalid[i]) return Calc(null, HistorianStatusCodes.BadAggregateInvalidInputs);
|
||||
if (_usableTicks[i] == 0)
|
||||
return _badTicks[i] > 0 ? Calc(null, HistorianStatusCodes.Bad) : AggregateEvaluator.NoData(bucketStart, now);
|
||||
|
||||
var status = _usableTicks[i] == duration ? HistorianStatusCodes.Good : HistorianStatusCodes.UncertainDataSubNormal;
|
||||
return _aggregate switch
|
||||
{
|
||||
HistoryAggregateType.TimeAverage => Calc(_bridgedArea[i] / _bridgedTicks[i], status),
|
||||
HistoryAggregateType.Total => Calc(_bridgedArea[i] / _bridgedTicks[i] * (duration / TicksPerSecond), status),
|
||||
HistoryAggregateType.TimeAverage2 => Calc(_rawArea[i] / _usableTicks[i], status),
|
||||
HistoryAggregateType.Total2 => Calc(_rawArea[i] / TicksPerSecond, status),
|
||||
_ => throw new HistoryAggregateNotSupportedException(_aggregate),
|
||||
};
|
||||
}
|
||||
|
||||
// ------------------------------------------------------------------------------------------ Segments
|
||||
|
||||
private static bool IsNumeric(StoredSample s) =>
|
||||
s.Value.Kind is HistorianValueKind.Double or HistorianValueKind.Int64 or HistorianValueKind.Boolean;
|
||||
|
||||
/// <summary>Sloped between two usable samples only when both are Double/Int64; anything Boolean is stepped.</summary>
|
||||
private static bool IsSloped(StoredSample a, StoredSample b) =>
|
||||
a.Value.Kind is HistorianValueKind.Double or HistorianValueKind.Int64
|
||||
&& b.Value.Kind is HistorianValueKind.Double or HistorianValueKind.Int64;
|
||||
|
||||
private long BucketStartTicks(long i) => _start + i * _plan.IntervalTicks;
|
||||
|
||||
private long BucketEnd(long i) => Math.Min(BucketStartTicks(i) + _plan.IntervalTicks, _end);
|
||||
|
||||
private void AddRaw(StoredSample from, long toTicks, double toValue) =>
|
||||
Distribute(from, toTicks, toValue, (i, area, ticks) =>
|
||||
{
|
||||
_rawArea[i] += area;
|
||||
_usableTicks[i] += ticks;
|
||||
});
|
||||
|
||||
/// <summary>Counts <c>[from.t, toTicks)</c> as outage (Bad) time; no value, never marks a bucket invalid.</summary>
|
||||
private void AddBad(StoredSample from, long toTicks)
|
||||
{
|
||||
var lo = Math.Max(from.TimeUtc.Ticks, _start);
|
||||
var hi = Math.Min(toTicks, _end);
|
||||
if (lo >= hi || _plan.BucketCount == 0) return;
|
||||
for (var i = (lo - _start) / _plan.IntervalTicks; i < _plan.BucketCount; i++)
|
||||
{
|
||||
var bs = BucketStartTicks(i);
|
||||
if (bs >= hi) break;
|
||||
var a = Math.Max(lo, bs);
|
||||
var b = Math.Min(hi, BucketEnd(i));
|
||||
if (a < b) _badTicks[i] += b - a;
|
||||
}
|
||||
}
|
||||
|
||||
private void AddBridged(StoredSample from, long toTicks, double toValue) =>
|
||||
Distribute(from, toTicks, toValue, (i, area, ticks) =>
|
||||
{
|
||||
_bridgedArea[i] += area;
|
||||
_bridgedTicks[i] += ticks;
|
||||
});
|
||||
|
||||
/// <summary>
|
||||
/// Splits the linear segment <c>[from.t, toTicks)</c> (value <c>from.v → toValue</c>) at the bucket edges, clipped to
|
||||
/// the window, and hands each bucket its trapezoid area (value·ticks) and duration. A non-numeric start sample
|
||||
/// marks every bucket it touches invalid instead.
|
||||
/// </summary>
|
||||
private void Distribute(StoredSample from, long toTicks, double toValue, Action<long, double, long> add)
|
||||
{
|
||||
var t0 = from.TimeUtc.Ticks;
|
||||
var lo = Math.Max(t0, _start);
|
||||
var hi = Math.Min(toTicks, _end);
|
||||
if (lo >= hi || _plan.BucketCount == 0) return;
|
||||
|
||||
var numeric = IsNumeric(from);
|
||||
var v0 = from.Value.Number;
|
||||
var span = (double)(toTicks - t0);
|
||||
double ValueAt(long x) => toValue == v0 ? v0 : v0 + (toValue - v0) * ((x - t0) / span);
|
||||
|
||||
for (var i = (lo - _start) / _plan.IntervalTicks; i < _plan.BucketCount; i++)
|
||||
{
|
||||
var bs = BucketStartTicks(i);
|
||||
if (bs >= hi) break;
|
||||
var a = Math.Max(lo, bs);
|
||||
var b = Math.Min(hi, BucketEnd(i));
|
||||
if (a >= b) continue;
|
||||
if (!numeric) { _invalid[i] = true; continue; }
|
||||
add(i, (ValueAt(a) + ValueAt(b)) / 2d * (b - a), b - a);
|
||||
}
|
||||
}
|
||||
|
||||
// ------------------------------------------------------------------------------------------ Interpolative
|
||||
|
||||
/// <summary>
|
||||
/// The time of Interpolative point <paramref name="j"/> (ascending time). Forward: the bucket starts
|
||||
/// <c>start + j·interval</c>. Reverse (Part 13 anchors a reverse read at its later time, the request's start):
|
||||
/// <c>End − (n−1−j)·interval</c>, i.e. <c>End, End − interval, …</c> once <see cref="BucketPlan.Assemble"/> has
|
||||
/// reversed the order — so a reverse read 10:00 → 09:00 at 1 h yields the value at 10:00.
|
||||
/// </summary>
|
||||
private long PointTicks(long j) =>
|
||||
_plan.Reversed ? _end - (_plan.BucketCount - 1 - j) * _plan.IntervalTicks : BucketStartTicks(j);
|
||||
|
||||
private DateTime PointTime(long j) => new(PointTicks(j), DateTimeKind.Utc);
|
||||
|
||||
/// <summary>Classifies every unclassified point strictly before <paramref name="beforeTicks"/>.</summary>
|
||||
private void ClassifyPointsBefore(long beforeTicks)
|
||||
{
|
||||
if (_aggregate != HistoryAggregateType.Interpolative) return;
|
||||
for (; _nextPoint < _plan.BucketCount; _nextPoint++)
|
||||
{
|
||||
var a = PointTicks(_nextPoint);
|
||||
if (a >= beforeTicks) break;
|
||||
|
||||
if (_prevAny is not { } p)
|
||||
{
|
||||
_pending.Add(new PendingPoint(_nextPoint, a, PointQuality.NoPrior));
|
||||
}
|
||||
else if (p.IsUsable && p.TimeUtc.Ticks == a)
|
||||
{
|
||||
// A usable sample exactly at the interval start: its own value and status.
|
||||
_points[_nextPoint] = IsNumeric(p)
|
||||
? new DataValueSnapshot(p.Value.Number, p.Status, PointTime(_nextPoint), default)
|
||||
: InvalidPoint(_nextPoint);
|
||||
}
|
||||
else if (!p.IsUsable)
|
||||
{
|
||||
_pending.Add(new PendingPoint(_nextPoint, a, PointQuality.Degraded));
|
||||
}
|
||||
else
|
||||
{
|
||||
// Inside a segment that starts at a usable sample; its quality is only known once the next sample
|
||||
// arrives (beforeTicks is that sample's time, or MaxValue at Finish ⇒ extrapolated).
|
||||
_pending.Add(new PendingPoint(_nextPoint, a, PointQuality.OnSegment));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Resolves the pending bucket starts against the bridging pair <c>(p, next)</c>.</summary>
|
||||
private void ResolvePending(StoredSample? p, StoredSample? next)
|
||||
{
|
||||
if (_pending.Count == 0) return;
|
||||
foreach (var pt in _pending)
|
||||
{
|
||||
var bucketStart = PointTime(pt.Bucket);
|
||||
if (p is not { } prev || pt.Quality == PointQuality.NoPrior)
|
||||
{
|
||||
_points[pt.Bucket] = new DataValueSnapshot(null, HistorianStatusCodes.BadNoData, bucketStart, default);
|
||||
continue;
|
||||
}
|
||||
if (!IsNumeric(prev) || next is { } nn && !IsNumeric(nn))
|
||||
{
|
||||
_points[pt.Bucket] = InvalidPoint(pt.Bucket);
|
||||
continue;
|
||||
}
|
||||
|
||||
// Good only on a segment that runs straight from prev to next (no outage skipped, not extrapolated), and only
|
||||
// when the samples that define the value are Good: both ends when sloped, prev alone when stepped.
|
||||
double value;
|
||||
bool good;
|
||||
if (next is { } n && IsSloped(prev, n))
|
||||
{
|
||||
value = prev.Value.Number + (n.Value.Number - prev.Value.Number)
|
||||
* ((double)(pt.Ticks - prev.TimeUtc.Ticks) / (n.TimeUtc.Ticks - prev.TimeUtc.Ticks));
|
||||
good = pt.Quality == PointQuality.OnSegment && prev.IsGood && n.IsGood;
|
||||
}
|
||||
else
|
||||
{
|
||||
value = prev.Value.Number;
|
||||
good = pt.Quality == PointQuality.OnSegment && next is not null && prev.IsGood;
|
||||
}
|
||||
var status = good ? HistorianStatusCodes.Good : HistorianStatusCodes.UncertainDataSubNormal;
|
||||
_points[pt.Bucket] = new DataValueSnapshot(value, HistorianInfoBits.WithInterpolated(status), bucketStart, default);
|
||||
}
|
||||
_pending.Clear();
|
||||
}
|
||||
|
||||
/// <summary>Marks every pending on-segment point degraded: its segment ends at a Bad sample.</summary>
|
||||
private void DegradePending()
|
||||
{
|
||||
for (var k = 0; k < _pending.Count; k++)
|
||||
if (_pending[k].Quality == PointQuality.OnSegment)
|
||||
_pending[k] = _pending[k] with { Quality = PointQuality.Degraded };
|
||||
}
|
||||
|
||||
private DataValueSnapshot InvalidPoint(long bucket) =>
|
||||
new(null, HistorianStatusCodes.BadAggregateInvalidInputs, PointTime(bucket), default);
|
||||
}
|
||||
+29
@@ -0,0 +1,29 @@
|
||||
-- OtOpcUa SQL Server historian backend - schema version 2: OPC UA Part 11 modified-value history (Gitea #668).
|
||||
-- $(Schema) is substituted by SqlServerSchemaBootstrapper (identifier-validated) BEFORE the script is split on GO lines.
|
||||
-- Every statement is guarded so a re-run is a no-op.
|
||||
--
|
||||
-- One row per sample a HistoryUpdate inserted, replaced or deleted, carrying the PRIOR value (all NULL for an
|
||||
-- insert). Rows are written by the SAME statement that changes hist.Sample (OUTPUT ... INTO), so a failed update
|
||||
-- leaves no audit row and a committed one always has its row.
|
||||
-- WARNING: OUTPUT ... INTO forbids, on its target table: enabled triggers, either side of a FOREIGN KEY, and CHECK
|
||||
-- constraints. Do not add any of those to SampleModified, or every HistoryUpdate fails.
|
||||
IF OBJECT_ID(N'[$(Schema)].[SampleModified]', N'U') IS NULL
|
||||
CREATE TABLE [$(Schema)].[SampleModified] (
|
||||
TagId int NOT NULL,
|
||||
SourceTimeUtc datetime2(7) NOT NULL,
|
||||
ModifiedId bigint IDENTITY(1,1) NOT NULL,
|
||||
ModificationTimeUtc datetime2(7) NOT NULL,
|
||||
UpdateType tinyint NOT NULL,
|
||||
Principal nvarchar(200) NOT NULL,
|
||||
ApiKeyId nvarchar(200) NULL,
|
||||
ValueKind tinyint NULL,
|
||||
NumValue float NULL,
|
||||
IntValue bigint NULL,
|
||||
StrValue nvarchar(max) NULL,
|
||||
StatusCode int NULL,
|
||||
ServerTimeUtc datetime2(7) NULL,
|
||||
CONSTRAINT PK_SampleModified PRIMARY KEY CLUSTERED (TagId, SourceTimeUtc, ModifiedId))
|
||||
GO
|
||||
IF NOT EXISTS (SELECT 1 FROM [$(Schema)].[SchemaVersion] WHERE Version = 2)
|
||||
INSERT [$(Schema)].[SchemaVersion] (Version, AppliedUtc) VALUES (2, SYSUTCDATETIME())
|
||||
GO
|
||||
@@ -91,6 +91,36 @@ internal static class SqlAggregateQueries
|
||||
"SELECT st.*, r.NumValue, r.SourceTimeUtc " +
|
||||
"FROM st LEFT JOIN r ON r.b = st.b AND r.rn = 1 ORDER BY st.b;";
|
||||
|
||||
/// <summary>
|
||||
/// Time-weighted family (#671: TimeAverage / TimeAverage2 / Total / Total2 / Interpolative) — <b>not</b> pushed down.
|
||||
/// One batch of five result sets, each <c>SourceTimeUtc, ValueKind, NumValue, IntValue, StrValue, StatusCode</c>:
|
||||
/// (1) the latest usable sample before <c>@start</c>, (2) the latest sample of any quality before <c>@start</c>,
|
||||
/// (3) the samples in <c>[@start, @end)</c> ascending, at most <c>@maxRows</c> (the raw-row cap + 1), (4) the first sample of any quality at or after
|
||||
/// <c>@end</c>, (5) the first usable one at or after <c>@end</c>. Each bound is a <c>TOP 1</c> seek on the
|
||||
/// <c>(TagId, SourceTimeUtc)</c> key. The caller streams them, in that order, into
|
||||
/// <see cref="TimeWeightedAggregator"/> (shared with QuestDB, so both backends agree exactly).
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Trade-off: the window's rows travel to the server instead of one row per bucket. Part 13's per-segment
|
||||
/// interpolation, bucket-edge splitting and the two bounding-value definitions would otherwise have to be written
|
||||
/// twice (T-SQL and QuestDB SQL) and kept bit-identical; evaluating once in C# keeps the backends equal by
|
||||
/// construction. Memory stays O(buckets): the rows are streamed, never materialised.
|
||||
/// </remarks>
|
||||
/// <param name="sample">Schema-qualified <c>Sample</c> table name.</param>
|
||||
/// <returns>The SQL batch.</returns>
|
||||
public static string TimeWeighted(string sample)
|
||||
{
|
||||
const string cols = "SourceTimeUtc, ValueKind, NumValue, IntValue, StrValue, StatusCode";
|
||||
const string usable = " AND StatusCode >= 0 AND ValueKind <> 0";
|
||||
var key = $"FROM {sample} WHERE TagId = @id AND ";
|
||||
return
|
||||
$"SELECT TOP 1 {cols} {key}SourceTimeUtc < @start{usable} ORDER BY SourceTimeUtc DESC; " +
|
||||
$"SELECT TOP 1 {cols} {key}SourceTimeUtc < @start ORDER BY SourceTimeUtc DESC; " +
|
||||
$"SELECT TOP (@maxRows) {cols} {key}SourceTimeUtc >= @start AND SourceTimeUtc < @end ORDER BY SourceTimeUtc ASC; " +
|
||||
$"SELECT TOP 1 {cols} {key}SourceTimeUtc >= @end ORDER BY SourceTimeUtc ASC; " +
|
||||
$"SELECT TOP 1 {cols} {key}SourceTimeUtc >= @end{usable} ORDER BY SourceTimeUtc ASC;";
|
||||
}
|
||||
|
||||
/// <summary>The <c>s</c> CTE: the window's samples with bucket index and the usable/good flags.</summary>
|
||||
private static string Classified(string sample) =>
|
||||
"WITH s AS (SELECT (DATEDIFF_BIG(microsecond, @start, SourceTimeUtc) * 10 " +
|
||||
|
||||
+140
-3
@@ -59,6 +59,7 @@ public sealed class SqlServerHistorianDataSource : IHistorianDataSource
|
||||
private readonly TimeProvider _time;
|
||||
|
||||
private readonly HistorianHealthCounters _health = new();
|
||||
private readonly int _maxTimeWeightedRawRows;
|
||||
|
||||
/// <summary>Creates the data source over the provider's shared connection, bootstrapper and tag catalog.</summary>
|
||||
/// <param name="conn">Connection helper.</param>
|
||||
@@ -66,10 +67,12 @@ public sealed class SqlServerHistorianDataSource : IHistorianDataSource
|
||||
/// <param name="catalog">Tag catalog (shared; reads use only its read-only lookup).</param>
|
||||
/// <param name="log">Optional logger.</param>
|
||||
/// <param name="timeProvider">Clock for "now" (AtTime beyond the last sample, server timestamps); system by default.</param>
|
||||
/// <param name="maxTimeWeightedRawRows">Raw in-window row cap of one time-weighted read (<= 0 = unbounded).</param>
|
||||
internal SqlServerHistorianDataSource(
|
||||
SqlHistorianConnection conn, SqlServerSchemaBootstrapper bootstrap, SqlServerTagCatalog catalog, ILogger? log = null,
|
||||
TimeProvider? timeProvider = null)
|
||||
TimeProvider? timeProvider = null, int maxTimeWeightedRawRows = TimeWeightedAggregator.DefaultMaxWindowRows)
|
||||
{
|
||||
_maxTimeWeightedRawRows = maxTimeWeightedRawRows;
|
||||
_conn = conn ?? throw new ArgumentNullException(nameof(conn));
|
||||
_bootstrap = bootstrap ?? throw new ArgumentNullException(nameof(bootstrap));
|
||||
_catalog = catalog ?? throw new ArgumentNullException(nameof(catalog));
|
||||
@@ -132,6 +135,97 @@ public sealed class SqlServerHistorianDataSource : IHistorianDataSource
|
||||
}
|
||||
}
|
||||
|
||||
// ------------------------------------------------------------------------------------------ Modified (Part 11, #668)
|
||||
|
||||
/// <inheritdoc />
|
||||
/// <remarks>
|
||||
/// Reads <c>SampleModified</c> (schema v2), the trail <see cref="SqlServerHistorianDataUpdater"/> writes in the
|
||||
/// same statement as each change. Windows follow <see cref="ReadRawAsync"/>; rows are ordered by
|
||||
/// <c>(SourceTimeUtc, ModifiedId)</c> (the clustered key after <c>TagId</c>, so a seek). <c>maxValues + 1</c> rows
|
||||
/// are fetched to report <see cref="HistoryModifiedReadResult.HasMore"/> exactly. An insert's audit row carries no
|
||||
/// prior value and is served <c>null</c> / <c>BadNoData</c>; its server timestamp is the modification time. An
|
||||
/// unknown tag is an empty result.
|
||||
/// </remarks>
|
||||
public async Task<HistoryModifiedReadResult> ReadModifiedAsync(
|
||||
string fullReference, DateTime startUtc, DateTime endUtc, uint maxValues, HistoryModifiedCursor? resumeAfter,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(fullReference);
|
||||
var bounded = maxValues > 0 && maxValues < int.MaxValue;
|
||||
try
|
||||
{
|
||||
var values = new List<HistoryModifiedValue>();
|
||||
var hasMore = false;
|
||||
var tagId = await ResolveAsync(fullReference, cancellationToken).ConfigureAwait(false);
|
||||
if (tagId is { } id)
|
||||
{
|
||||
var reverse = startUtc > endUtc;
|
||||
string where;
|
||||
if (startUtc == endUtc) where = "SourceTimeUtc = @lo";
|
||||
else if (!reverse) where = "SourceTimeUtc >= @lo AND SourceTimeUtc < @hi";
|
||||
else where = "SourceTimeUtc > @lo AND SourceTimeUtc <= @hi";
|
||||
if (resumeAfter is not null)
|
||||
where += reverse
|
||||
? " AND (SourceTimeUtc < @ct OR (SourceTimeUtc = @ct AND ModifiedId < @cid))"
|
||||
: " AND (SourceTimeUtc > @ct OR (SourceTimeUtc = @ct AND ModifiedId > @cid))";
|
||||
var order = reverse ? "DESC" : "ASC";
|
||||
var (lo, hi) = startUtc <= endUtc ? (startUtc, endUtc) : (endUtc, startUtc);
|
||||
|
||||
await using var conn = await _conn.OpenAsync(cancellationToken).ConfigureAwait(false);
|
||||
await using var cmd = _conn.CreateCommand(conn,
|
||||
$"SELECT {(bounded ? "TOP (@take) " : "")}SourceTimeUtc, ValueKind, NumValue, IntValue, StrValue, StatusCode, " +
|
||||
"ServerTimeUtc, ModifiedId, ModificationTimeUtc, UpdateType, Principal " +
|
||||
$"FROM {_conn.Q("SampleModified")} WHERE TagId = @id AND {where} " +
|
||||
$"ORDER BY SourceTimeUtc {order}, ModifiedId {order}");
|
||||
if (bounded) cmd.Parameters.Add(new SqlParameter("@take", SqlDbType.BigInt) { Value = (long)maxValues + 1 });
|
||||
cmd.Parameters.Add(new SqlParameter("@id", SqlDbType.Int) { Value = id });
|
||||
SqlHistorianConnection.AddDateTime2(cmd, "@lo", lo);
|
||||
SqlHistorianConnection.AddDateTime2(cmd, "@hi", hi);
|
||||
if (resumeAfter is not null)
|
||||
{
|
||||
SqlHistorianConnection.AddDateTime2(cmd, "@ct", resumeAfter.SourceTimeUtc);
|
||||
cmd.Parameters.Add(new SqlParameter("@cid", SqlDbType.BigInt) { Value = resumeAfter.ModifiedId });
|
||||
}
|
||||
|
||||
await using var reader = await cmd.ExecuteReaderAsync(cancellationToken).ConfigureAwait(false);
|
||||
while (await reader.ReadAsync(cancellationToken).ConfigureAwait(false))
|
||||
{
|
||||
if (bounded && values.Count == maxValues)
|
||||
{
|
||||
hasMore = true;
|
||||
break;
|
||||
}
|
||||
|
||||
var sourceTime = SqlHistorianConnection.GetUtc(reader, 0);
|
||||
var modifiedAt = SqlHistorianConnection.GetUtc(reader, 8);
|
||||
DataValueSnapshot prior;
|
||||
if (reader.IsDBNull(1))
|
||||
{
|
||||
// An insert: there was no prior value.
|
||||
prior = new DataValueSnapshot(null, HistorianStatusCodes.BadNoData, sourceTime, modifiedAt);
|
||||
}
|
||||
else
|
||||
{
|
||||
var s = ReadStored(reader, 0);
|
||||
prior = new DataValueSnapshot(s.Value.ToObject(), s.Status, s.TimeUtc,
|
||||
reader.IsDBNull(6) ? modifiedAt : SqlHistorianConnection.GetUtc(reader, 6));
|
||||
}
|
||||
|
||||
values.Add(new HistoryModifiedValue(
|
||||
prior, modifiedAt, (HistoryModificationType)reader.GetByte(9), reader.GetString(10), reader.GetInt64(7), sourceTime));
|
||||
}
|
||||
}
|
||||
|
||||
RecordSuccess();
|
||||
return new HistoryModifiedReadResult(values, hasMore);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
RecordFailure(ex);
|
||||
throw;
|
||||
}
|
||||
}
|
||||
|
||||
// ------------------------------------------------------------------------------------------ AtTime
|
||||
|
||||
/// <inheritdoc />
|
||||
@@ -298,10 +392,12 @@ public sealed class SqlServerHistorianDataSource : IHistorianDataSource
|
||||
/// <para>
|
||||
/// Values: Count is <see cref="int"/>; Start/End keep the sample's type; every other aggregate is
|
||||
/// <see cref="double"/> computed over <c>NumValue</c> (Boolean = 0/1). These are the simple per-sample
|
||||
/// definitions, not Part 13's interval-boundary (interpolated bound / time-weighted) semantics.
|
||||
/// definitions. The Part 13 time-weighted family (TimeAverage, TimeAverage2, Total, Total2, Interpolative, #671) is
|
||||
/// not pushed down: the window's rows plus four bounding rows are streamed into the shared
|
||||
/// <see cref="TimeWeightedAggregator"/>, which fills every interval (an empty one between bounds gets a value).
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
/// <exception cref="HistoryAggregateNotSupportedException">The aggregate is not one of the thirteen served.</exception>
|
||||
/// <exception cref="HistoryAggregateNotSupportedException">The aggregate is not one of the seventeen served.</exception>
|
||||
/// <exception cref="ArgumentOutOfRangeException">The window holds more than <see cref="MaxProcessedBuckets"/> intervals.</exception>
|
||||
public async Task<HistoryReadResult> ReadProcessedAsync(
|
||||
string fullReference, DateTime startUtc, DateTime endUtc, TimeSpan interval,
|
||||
@@ -316,6 +412,14 @@ public sealed class SqlServerHistorianDataSource : IHistorianDataSource
|
||||
var now = _time.GetUtcNow().UtcDateTime;
|
||||
var evaluated = new Dictionary<long, DataValueSnapshot>();
|
||||
var tagId = plan.BucketCount == 0 ? null : await ResolveAsync(fullReference, cancellationToken).ConfigureAwait(false);
|
||||
if (family == AggregateFamily.TimeWeighted)
|
||||
{
|
||||
var tw = new TimeWeightedAggregator(aggregate, plan);
|
||||
if (tagId is { } twId) await StreamTimeWeightedAsync(twId, fullReference, plan, tw, cancellationToken).ConfigureAwait(false);
|
||||
var twResult = tw.Finish(now);
|
||||
RecordSuccess();
|
||||
return new HistoryReadResult(twResult, ContinuationPoint: null);
|
||||
}
|
||||
if (tagId is { } id)
|
||||
{
|
||||
var sample = _conn.Q("Sample");
|
||||
@@ -344,6 +448,11 @@ public sealed class SqlServerHistorianDataSource : IHistorianDataSource
|
||||
RecordSuccess();
|
||||
return new HistoryReadResult(result, ContinuationPoint: null);
|
||||
}
|
||||
catch (HistoryReadLimitExceededException)
|
||||
{
|
||||
// A client-request limit (the raw-row cap), not a backend fault: not counted as a failure.
|
||||
throw;
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
RecordFailure(ex);
|
||||
@@ -351,6 +460,34 @@ public sealed class SqlServerHistorianDataSource : IHistorianDataSource
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Streams the time-weighted batch (<see cref="SqlAggregateQueries.TimeWeighted"/>) into <paramref name="tw"/>: the
|
||||
/// two before-bounds, the window, the two after-bounds — ascending, with duplicates dropped by the aggregator.
|
||||
/// </summary>
|
||||
private async Task StreamTimeWeightedAsync(int id, string tag, BucketPlan plan, TimeWeightedAggregator tw, CancellationToken ct)
|
||||
{
|
||||
await using var conn = await _conn.OpenAsync(ct).ConfigureAwait(false);
|
||||
await using var cmd = _conn.CreateCommand(conn, SqlAggregateQueries.TimeWeighted(_conn.Q("Sample")));
|
||||
cmd.Parameters.Add(new SqlParameter("@id", SqlDbType.Int) { Value = id });
|
||||
SqlHistorianConnection.AddDateTime2(cmd, "@start", DateTime.SpecifyKind(plan.StartUtc, DateTimeKind.Utc));
|
||||
SqlHistorianConnection.AddDateTime2(cmd, "@end", DateTime.SpecifyKind(plan.EndUtc, DateTimeKind.Utc));
|
||||
cmd.Parameters.Add(new SqlParameter("@maxRows", SqlDbType.Int)
|
||||
{ Value = TimeWeightedAggregator.WindowFetchLimit(_maxTimeWeightedRawRows) });
|
||||
await using var reader = await cmd.ExecuteReaderAsync(ct).ConfigureAwait(false);
|
||||
var resultSet = 0;
|
||||
do
|
||||
{
|
||||
long windowRows = 0;
|
||||
while (await reader.ReadAsync(ct).ConfigureAwait(false))
|
||||
{
|
||||
// Result set 2 is the window (TOP (@maxRows) = cap + 1): one row past the cap fails the node.
|
||||
if (resultSet == 2) TimeWeightedAggregator.CheckWindowRows(++windowRows, _maxTimeWeightedRawRows, tag);
|
||||
tw.Add(ReadStored(reader, 0));
|
||||
}
|
||||
resultSet++;
|
||||
} while (await reader.NextResultAsync(ct).ConfigureAwait(false));
|
||||
}
|
||||
|
||||
private static BucketRow ReadBucket(SqlDataReader reader, AggregateFamily family)
|
||||
{
|
||||
static double? D(SqlDataReader r, int i) => r.IsDBNull(i) ? null : r.GetDouble(i);
|
||||
|
||||
+54
-8
@@ -22,6 +22,11 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer;
|
||||
/// is written.</para>
|
||||
/// <para><b>Unknown tag.</b> Insert/Update create it via <see cref="SqlServerTagCatalog"/>; Replace and the
|
||||
/// deletes report <c>BadNoEntryExists</c> (DeleteRaw: <c>BadNoData</c>) without creating anything.</para>
|
||||
/// <para><b>Modified-value history (Part 11, #668).</b> Every row the MERGE inserts or updates and every row a
|
||||
/// delete removes is copied into <c>SampleModified</c> by an <c>OUTPUT … INTO</c> on the SAME statement — the prior
|
||||
/// value from <c>deleted.*</c> (NULL for an insert), the caller's <see cref="HistoryUpdateContext"/>, the
|
||||
/// <see cref="HistoryModificationType"/> and <c>SYSUTCDATETIME()</c>. One statement means the sample change and its
|
||||
/// audit row commit or roll back together: a failed update leaves no audit row.</para>
|
||||
/// </summary>
|
||||
internal sealed class SqlServerHistorianDataUpdater : IHistorianDataUpdater
|
||||
{
|
||||
@@ -39,10 +44,11 @@ internal sealed class SqlServerHistorianDataUpdater : IHistorianDataUpdater
|
||||
|
||||
/// <inheritdoc/>
|
||||
public async Task<IReadOnlyList<uint>> UpdateDataAsync(string tag, HistoryUpdateMode mode,
|
||||
IReadOnlyList<HistorizationValue> values, CancellationToken ct)
|
||||
IReadOnlyList<HistorizationValue> values, HistoryUpdateContext context, CancellationToken ct)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(tag);
|
||||
ArgumentNullException.ThrowIfNull(values);
|
||||
ArgumentNullException.ThrowIfNull(context);
|
||||
var plan = HistoryUpdatePlanner.Plan(tag, mode, values);
|
||||
if (plan.Winners.Count == 0) return HistoryUpdatePlanner.Complete(plan, mode, new Dictionary<long, UpsertOutcome>());
|
||||
|
||||
@@ -52,14 +58,16 @@ internal sealed class SqlServerHistorianDataUpdater : IHistorianDataUpdater
|
||||
|
||||
var outcome = tagId is null
|
||||
? new Dictionary<long, UpsertOutcome>()
|
||||
: await MergeAsync(tagId.Value, mode, plan.Winners.Select(w => w.Row).ToList(), ct).ConfigureAwait(false);
|
||||
: await MergeAsync(tagId.Value, mode, plan.Winners.Select(w => w.Row).ToList(), context, ct).ConfigureAwait(false);
|
||||
return HistoryUpdatePlanner.Complete(plan, mode, outcome);
|
||||
}
|
||||
|
||||
/// <inheritdoc/>
|
||||
public async Task<uint> DeleteRawAsync(string tag, DateTime startUtc, DateTime endUtc, CancellationToken ct)
|
||||
public async Task<uint> DeleteRawAsync(string tag, DateTime startUtc, DateTime endUtc, HistoryUpdateContext context,
|
||||
CancellationToken ct)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(tag);
|
||||
ArgumentNullException.ThrowIfNull(context);
|
||||
if (!HistorianTagNames.IsValid(tag)) return HistorianStatusCodes.BadInvalidArgument;
|
||||
var tagId = await _tags.TryGetTagIdAsync(tag, ct).ConfigureAwait(false);
|
||||
if (tagId is null) return HistoryUpdatePlanner.DeleteRawResult(tagKnown: false, 0);
|
||||
@@ -67,9 +75,11 @@ internal sealed class SqlServerHistorianDataUpdater : IHistorianDataUpdater
|
||||
await using var tx = (SqlTransaction)await conn.BeginTransactionAsync(ct).ConfigureAwait(false);
|
||||
int rows;
|
||||
await using (var cmd = _conn.CreateCommand(conn,
|
||||
$"DELETE FROM {_conn.Q("Sample")} WHERE TagId = @tagId AND SourceTimeUtc >= @s AND SourceTimeUtc < @e", tx))
|
||||
$"DELETE FROM {_conn.Q("Sample")} {AuditInto(HistoryModificationType.Delete)} " +
|
||||
"WHERE TagId = @tagId AND SourceTimeUtc >= @s AND SourceTimeUtc < @e", tx))
|
||||
{
|
||||
cmd.Parameters.Add(new SqlParameter("@tagId", System.Data.SqlDbType.Int) { Value = tagId.Value });
|
||||
AddAuditParameters(cmd, context);
|
||||
SqlHistorianConnection.AddDateTime2(cmd, "@s", startUtc);
|
||||
SqlHistorianConnection.AddDateTime2(cmd, "@e", endUtc);
|
||||
rows = await cmd.ExecuteNonQueryAsync(ct).ConfigureAwait(false);
|
||||
@@ -79,10 +89,12 @@ internal sealed class SqlServerHistorianDataUpdater : IHistorianDataUpdater
|
||||
}
|
||||
|
||||
/// <inheritdoc/>
|
||||
public async Task<IReadOnlyList<uint>> DeleteAtTimeAsync(string tag, IReadOnlyList<DateTime> timesUtc, CancellationToken ct)
|
||||
public async Task<IReadOnlyList<uint>> DeleteAtTimeAsync(string tag, IReadOnlyList<DateTime> timesUtc,
|
||||
HistoryUpdateContext context, CancellationToken ct)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(tag);
|
||||
ArgumentNullException.ThrowIfNull(timesUtc);
|
||||
ArgumentNullException.ThrowIfNull(context);
|
||||
if (!HistorianTagNames.IsValid(tag))
|
||||
{
|
||||
var refused = new uint[timesUtc.Count];
|
||||
@@ -100,9 +112,11 @@ internal sealed class SqlServerHistorianDataUpdater : IHistorianDataUpdater
|
||||
await using var conn = await _conn.OpenAsync(ct).ConfigureAwait(false);
|
||||
await using var tx = (SqlTransaction)await conn.BeginTransactionAsync(ct).ConfigureAwait(false);
|
||||
await using (var cmd = _conn.CreateCommand(conn,
|
||||
$"DELETE s OUTPUT deleted.SourceTimeUtc FROM {_conn.Q("Sample")} AS s JOIN @times AS t ON s.SourceTimeUtc = t.T WHERE s.TagId = @tagId", tx))
|
||||
$"DELETE s {AuditInto(HistoryModificationType.Delete)} OUTPUT deleted.SourceTimeUtc " +
|
||||
$"FROM {_conn.Q("Sample")} AS s JOIN @times AS t ON s.SourceTimeUtc = t.T WHERE s.TagId = @tagId", tx))
|
||||
{
|
||||
cmd.Parameters.Add(new SqlParameter("@tagId", System.Data.SqlDbType.Int) { Value = tagId.Value });
|
||||
AddAuditParameters(cmd, context);
|
||||
_conn.AddStructured(cmd, "@times", "TimeRow",
|
||||
SampleRowMapper.ToTimeRecords(distinct.Select(t => new DateTime(t, DateTimeKind.Utc))));
|
||||
await using var rd = await cmd.ExecuteReaderAsync(ct).ConfigureAwait(false);
|
||||
@@ -114,7 +128,7 @@ internal sealed class SqlServerHistorianDataUpdater : IHistorianDataUpdater
|
||||
}
|
||||
|
||||
private async Task<Dictionary<long, UpsertOutcome>> MergeAsync(int tagId, HistoryUpdateMode mode,
|
||||
List<HistorianSampleRow> rows, CancellationToken ct)
|
||||
List<HistorianSampleRow> rows, HistoryUpdateContext context, CancellationToken ct)
|
||||
{
|
||||
var sql = $"MERGE {_conn.Q("Sample")} WITH (HOLDLOCK) AS t USING @rows AS r ON t.TagId = @tagId AND t.SourceTimeUtc = r.SourceTimeUtc ";
|
||||
if (mode != HistoryUpdateMode.Insert)
|
||||
@@ -123,7 +137,9 @@ internal sealed class SqlServerHistorianDataUpdater : IHistorianDataUpdater
|
||||
if (mode != HistoryUpdateMode.Replace)
|
||||
sql += "WHEN NOT MATCHED THEN INSERT (TagId, SourceTimeUtc, ValueKind, NumValue, IntValue, StrValue, StatusCode, ServerTimeUtc) " +
|
||||
"VALUES (@tagId, r.SourceTimeUtc, r.ValueKind, r.NumValue, r.IntValue, r.StrValue, r.StatusCode, SYSUTCDATETIME()) ";
|
||||
sql += "OUTPUT $action, r.SourceTimeUtc;";
|
||||
// The MERGE's deleted.* is NULL for an inserted row, so one OUTPUT INTO records both an insert (no prior value)
|
||||
// and a replace (the prior value). TagId comes from inserted.* — present on both branches.
|
||||
sql += $"{AuditInto(ToModificationType(mode), "inserted")} OUTPUT $action, r.SourceTimeUtc;";
|
||||
|
||||
var outcome = new Dictionary<long, UpsertOutcome>();
|
||||
await using var conn = await _conn.OpenAsync(ct).ConfigureAwait(false);
|
||||
@@ -131,6 +147,7 @@ internal sealed class SqlServerHistorianDataUpdater : IHistorianDataUpdater
|
||||
await using (var cmd = _conn.CreateCommand(conn, sql, tx))
|
||||
{
|
||||
cmd.Parameters.Add(new SqlParameter("@tagId", System.Data.SqlDbType.Int) { Value = tagId });
|
||||
AddAuditParameters(cmd, context);
|
||||
_conn.AddStructured(cmd, "@rows", "SampleRow", SampleRowMapper.ToSampleRecords(rows));
|
||||
await using var rd = await cmd.ExecuteReaderAsync(ct).ConfigureAwait(false);
|
||||
while (await rd.ReadAsync(ct).ConfigureAwait(false))
|
||||
@@ -141,4 +158,33 @@ internal sealed class SqlServerHistorianDataUpdater : IHistorianDataUpdater
|
||||
await tx.CommitAsync(ct).ConfigureAwait(false);
|
||||
return outcome;
|
||||
}
|
||||
|
||||
/// <summary>The Part 11 modification type recorded for an update mode.</summary>
|
||||
/// <param name="mode">The update mode.</param>
|
||||
/// <returns>Insert ⇒ Insert, Replace ⇒ Replace, Update ⇒ Update.</returns>
|
||||
internal static HistoryModificationType ToModificationType(HistoryUpdateMode mode) => mode switch
|
||||
{
|
||||
HistoryUpdateMode.Insert => HistoryModificationType.Insert,
|
||||
HistoryUpdateMode.Replace => HistoryModificationType.Replace,
|
||||
_ => HistoryModificationType.Update,
|
||||
};
|
||||
|
||||
/// <summary>
|
||||
/// The <c>OUTPUT … INTO SampleModified</c> clause copying each affected row's prior value (<c>deleted.*</c>) into
|
||||
/// the modification trail. <paramref name="keySource"/> names the pseudo-table that carries <c>TagId</c> /
|
||||
/// <c>SourceTimeUtc</c> for every affected row (<c>deleted</c> for a DELETE, <c>inserted</c> for the MERGE).
|
||||
/// </summary>
|
||||
private string AuditInto(HistoryModificationType type, string keySource = "deleted") =>
|
||||
$"OUTPUT {keySource}.TagId, {keySource}.SourceTimeUtc, SYSUTCDATETIME(), {(byte)type}, @principal, @apiKeyId, " +
|
||||
"deleted.ValueKind, deleted.NumValue, deleted.IntValue, deleted.StrValue, deleted.StatusCode, deleted.ServerTimeUtc " +
|
||||
$"INTO {_conn.Q("SampleModified")} (TagId, SourceTimeUtc, ModificationTimeUtc, UpdateType, Principal, ApiKeyId, " +
|
||||
"ValueKind, NumValue, IntValue, StrValue, StatusCode, ServerTimeUtc)";
|
||||
|
||||
private static void AddAuditParameters(SqlCommand cmd, HistoryUpdateContext context)
|
||||
{
|
||||
cmd.Parameters.Add(new SqlParameter("@principal", System.Data.SqlDbType.NVarChar, HistoryUpdateContext.MaxIdentityLength)
|
||||
{ Value = context.StoredPrincipal });
|
||||
cmd.Parameters.Add(new SqlParameter("@apiKeyId", System.Data.SqlDbType.NVarChar, HistoryUpdateContext.MaxIdentityLength)
|
||||
{ Value = (object?)context.StoredApiKeyId ?? DBNull.Value });
|
||||
}
|
||||
}
|
||||
|
||||
+23
-1
@@ -18,10 +18,26 @@ public sealed partial class SqlServerHistorianOptions
|
||||
|
||||
/// <summary>
|
||||
/// <c>true</c> creates the schema and applies missing scripts on first use; <c>false</c> only verifies
|
||||
/// that schema version 1 exists and fails every historian call with a clear error otherwise.
|
||||
/// that the required schema version (<c>SqlServerSchemaBootstrapper.RequiredVersion</c>, currently 2) exists and
|
||||
/// fails every historian call with a clear error otherwise.
|
||||
/// </summary>
|
||||
public bool AutoCreateSchema { get; set; } = true;
|
||||
|
||||
/// <summary>
|
||||
/// Days of history to keep, measured on source time. <c>null</c> (default) keeps everything. When set,
|
||||
/// a driver node deletes older <c>hist.Sample</c>/<c>hist.Event</c> rows every <see cref="PurgeInterval"/>.
|
||||
/// </summary>
|
||||
public int? RetentionDays { get; set; }
|
||||
|
||||
/// <summary>How often the retention purge runs. Must be positive.</summary>
|
||||
public TimeSpan PurgeInterval { get; set; } = TimeSpan.FromHours(1);
|
||||
|
||||
/// <summary>
|
||||
/// Rows per <c>DELETE TOP(n)</c> batch. Default 4000 stays under SQL Server's 5000-lock escalation
|
||||
/// threshold so a purge never escalates to a table lock.
|
||||
/// </summary>
|
||||
public int PurgeBatchSize { get; set; } = 4000;
|
||||
|
||||
[GeneratedRegex("^[A-Za-z_][A-Za-z0-9_]{0,127}$")]
|
||||
private static partial Regex IdentifierRegex();
|
||||
|
||||
@@ -41,6 +57,12 @@ public sealed partial class SqlServerHistorianOptions
|
||||
errors.Add("ServerHistorian:SqlServer:Schema must be a SQL identifier matching ^[A-Za-z_][A-Za-z0-9_]{0,127}$.");
|
||||
if (CommandTimeout <= TimeSpan.Zero)
|
||||
errors.Add("ServerHistorian:SqlServer:CommandTimeout must be greater than zero.");
|
||||
if (RetentionDays is < 1)
|
||||
errors.Add("ServerHistorian:SqlServer:RetentionDays must be at least 1 when set (omit it to keep history forever).");
|
||||
if (PurgeInterval <= TimeSpan.Zero)
|
||||
errors.Add("ServerHistorian:SqlServer:PurgeInterval must be greater than zero.");
|
||||
if (PurgeBatchSize < 1)
|
||||
errors.Add("ServerHistorian:SqlServer:PurgeBatchSize must be at least 1.");
|
||||
return errors;
|
||||
}
|
||||
}
|
||||
|
||||
+17
-2
@@ -1,3 +1,4 @@
|
||||
using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common;
|
||||
using System.Collections.Frozen;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
@@ -29,11 +30,16 @@ public sealed class SqlServerHistorianProvider : IHistorianBackendProvider
|
||||
|
||||
/// <summary>Creates the provider over the <c>ServerHistorian:SqlServer</c> options (validated elsewhere).</summary>
|
||||
/// <param name="options">The <c>ServerHistorian:SqlServer</c> options.</param>
|
||||
public SqlServerHistorianProvider(SqlServerHistorianOptions options)
|
||||
/// <param name="maxTimeWeightedRawRows">The shared <c>ServerHistorian:MaxTimeWeightedRawRows</c> cap (<= 0 = unbounded).</param>
|
||||
public SqlServerHistorianProvider(SqlServerHistorianOptions options,
|
||||
int maxTimeWeightedRawRows = TimeWeightedAggregator.DefaultMaxWindowRows)
|
||||
{
|
||||
_options = options ?? throw new ArgumentNullException(nameof(options));
|
||||
_maxTimeWeightedRawRows = maxTimeWeightedRawRows;
|
||||
}
|
||||
|
||||
private readonly int _maxTimeWeightedRawRows;
|
||||
|
||||
/// <inheritdoc />
|
||||
public string Name => "SqlServer";
|
||||
|
||||
@@ -63,7 +69,7 @@ public sealed class SqlServerHistorianProvider : IHistorianBackendProvider
|
||||
{
|
||||
var s = Get(sp);
|
||||
return new SqlServerHistorianDataSource(s.Conn, s.Bootstrap, s.Catalog,
|
||||
Loggers(sp).CreateLogger<SqlServerHistorianDataSource>());
|
||||
Loggers(sp).CreateLogger<SqlServerHistorianDataSource>(), maxTimeWeightedRawRows: _maxTimeWeightedRawRows);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
@@ -96,5 +102,14 @@ public sealed class SqlServerHistorianProvider : IHistorianBackendProvider
|
||||
return new SqlServerHistorianDataUpdater(s.Conn, s.Catalog);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public IHistorianRetention? CreateRetention(IServiceProvider sp)
|
||||
{
|
||||
if (_options.RetentionDays is not { } days) return null;
|
||||
var s = Get(sp);
|
||||
return new SqlServerHistorianRetention(s.Conn, s.Bootstrap, _options.PurgeInterval, days,
|
||||
_options.PurgeBatchSize, Loggers(sp).CreateLogger<SqlServerHistorianRetention>());
|
||||
}
|
||||
|
||||
private sealed record Shared(SqlHistorianConnection Conn, SqlServerSchemaBootstrapper Bootstrap, SqlServerTagCatalog Catalog);
|
||||
}
|
||||
|
||||
+154
@@ -0,0 +1,154 @@
|
||||
using System.Data;
|
||||
using Microsoft.Data.SqlClient;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using ZB.MOM.WW.OtOpcUa.Runtime.Historian;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer;
|
||||
|
||||
/// <summary>
|
||||
/// Deletes <c>hist.Sample</c> / <c>hist.SampleModified</c> / <c>hist.Event</c> rows older than the retention window, in small
|
||||
/// <c>DELETE TOP(n)</c> batches (under the lock-escalation threshold). Samples are purged per tag so each
|
||||
/// delete is a seek on the clustered key <c>(TagId, SourceTimeUtc)</c>; events seek on <c>CX_Event_Time</c>.
|
||||
/// A session <c>sp_getapplock</c> (timeout 0) serialises purgers that share one database; a loser skips.
|
||||
/// To purge another table later, add it to <see cref="PurgeAsync"/>.
|
||||
/// </summary>
|
||||
internal sealed class SqlServerHistorianRetention : IHistorianRetention
|
||||
{
|
||||
private readonly SqlHistorianConnection _conn;
|
||||
private readonly SqlServerSchemaBootstrapper _bootstrap;
|
||||
private readonly int _days;
|
||||
private readonly int _batch;
|
||||
private readonly TimeProvider _time;
|
||||
private readonly ILogger _log;
|
||||
|
||||
public SqlServerHistorianRetention(SqlHistorianConnection conn, SqlServerSchemaBootstrapper bootstrap,
|
||||
TimeSpan interval, int retentionDays, int batchSize, ILogger? log = null, TimeProvider? time = null)
|
||||
{
|
||||
_conn = conn;
|
||||
_bootstrap = bootstrap;
|
||||
Interval = interval;
|
||||
_days = retentionDays;
|
||||
_batch = batchSize;
|
||||
_log = log ?? NullLogger.Instance;
|
||||
_time = time ?? TimeProvider.System;
|
||||
}
|
||||
|
||||
public TimeSpan Interval { get; }
|
||||
|
||||
/// <summary>Test hook invoked after every delete batch.</summary>
|
||||
internal Action? AfterBatch { get; set; }
|
||||
|
||||
/// <summary>Test seam for <see cref="SqlConnection.ClearPool"/>.</summary>
|
||||
internal Action<SqlConnection> ClearPool { get; set; } = SqlConnection.ClearPool;
|
||||
|
||||
public async Task<IReadOnlyList<RetentionPurgeResult>> PurgeAsync(CancellationToken ct)
|
||||
{
|
||||
await _bootstrap.EnsureAsync(ct).ConfigureAwait(false);
|
||||
var cutoff = _time.GetUtcNow().UtcDateTime.AddDays(-_days);
|
||||
|
||||
await using var conn = await _conn.OpenAsync(ct).ConfigureAwait(false);
|
||||
var acquired = false;
|
||||
try
|
||||
{
|
||||
try
|
||||
{
|
||||
acquired = await TryLockAsync(conn, ct).ConfigureAwait(false);
|
||||
}
|
||||
catch
|
||||
{
|
||||
// Ambiguous: the server may have granted the lock before the client saw the reply. We cannot
|
||||
// release what we are not sure we hold, so close the session physically instead.
|
||||
ClearPool(conn);
|
||||
throw;
|
||||
}
|
||||
|
||||
if (!acquired)
|
||||
{
|
||||
_log.LogInformation("Historian retention purge skipped: another purger holds the applock.");
|
||||
return [];
|
||||
}
|
||||
|
||||
long samples = 0, modified = 0;
|
||||
var tagIds = new List<int>();
|
||||
await using (var cmd = _conn.CreateCommand(conn, $"SELECT TagId FROM {_conn.Q("Tag")}"))
|
||||
await using (var r = await cmd.ExecuteReaderAsync(ct).ConfigureAwait(false))
|
||||
while (await r.ReadAsync(ct).ConfigureAwait(false)) tagIds.Add(r.GetInt32(0));
|
||||
|
||||
foreach (var id in tagIds)
|
||||
{
|
||||
samples += await DeleteLoopAsync(conn,
|
||||
$"DELETE TOP (@n) FROM {_conn.Q("Sample")} WHERE TagId = @id AND SourceTimeUtc < @cutoff",
|
||||
cutoff, id, ct).ConfigureAwait(false);
|
||||
// The Part 11 modification trail (#668) ages out on the SAME source-time cutoff as the samples it
|
||||
// describes; a purge never writes to it (no trigger, no OUTPUT INTO).
|
||||
modified += await DeleteLoopAsync(conn,
|
||||
$"DELETE TOP (@n) FROM {_conn.Q("SampleModified")} WHERE TagId = @id AND SourceTimeUtc < @cutoff",
|
||||
cutoff, id, ct).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
var events = await DeleteLoopAsync(conn,
|
||||
$"DELETE TOP (@n) FROM {_conn.Q("Event")} WHERE EventTimeUtc < @cutoff",
|
||||
cutoff, null, ct).ConfigureAwait(false);
|
||||
|
||||
return
|
||||
[
|
||||
new RetentionPurgeResult("Sample", samples),
|
||||
new RetentionPurgeResult("SampleModified", modified),
|
||||
new RetentionPurgeResult("Event", events),
|
||||
];
|
||||
}
|
||||
finally
|
||||
{
|
||||
// Only when WE hold it: releasing an applock we were refused fails, which would log a false Warning
|
||||
// and clear the shared pool on every skipped interval.
|
||||
if (acquired) await ReleaseAsync(conn).ConfigureAwait(false);
|
||||
}
|
||||
}
|
||||
|
||||
private async Task<long> DeleteLoopAsync(SqlConnection conn, string sql, DateTime cutoff, int? tagId, CancellationToken ct)
|
||||
{
|
||||
long total = 0;
|
||||
while (true)
|
||||
{
|
||||
ct.ThrowIfCancellationRequested();
|
||||
await using var cmd = _conn.CreateCommand(conn, sql);
|
||||
cmd.Parameters.Add(new SqlParameter("@n", SqlDbType.Int) { Value = _batch });
|
||||
if (tagId is { } id) cmd.Parameters.Add(new SqlParameter("@id", SqlDbType.Int) { Value = id });
|
||||
SqlHistorianConnection.AddDateTime2(cmd, "@cutoff", cutoff);
|
||||
var n = await cmd.ExecuteNonQueryAsync(ct).ConfigureAwait(false);
|
||||
total += n;
|
||||
AfterBatch?.Invoke();
|
||||
if (n < _batch) return total;
|
||||
}
|
||||
}
|
||||
|
||||
private string LockResource => $"OtOpcUa.Historian.Retention.{_conn.Schema}";
|
||||
|
||||
private async Task<bool> TryLockAsync(SqlConnection conn, CancellationToken ct)
|
||||
{
|
||||
await using var cmd = _conn.CreateCommand(conn,
|
||||
"DECLARE @r int; EXEC @r = sp_getapplock @Resource = @res, @LockMode = 'Exclusive', @LockOwner = 'Session', @LockTimeout = 0; SELECT @r;");
|
||||
cmd.Parameters.Add(new SqlParameter("@res", SqlDbType.NVarChar, 255) { Value = LockResource });
|
||||
return Convert.ToInt32(await cmd.ExecuteScalarAsync(ct).ConfigureAwait(false)) >= 0;
|
||||
}
|
||||
|
||||
private async Task ReleaseAsync(SqlConnection conn)
|
||||
{
|
||||
try
|
||||
{
|
||||
await using var cmd = _conn.CreateCommand(conn,
|
||||
"EXEC sp_releaseapplock @Resource = @res, @LockOwner = 'Session';");
|
||||
cmd.Parameters.Add(new SqlParameter("@res", SqlDbType.NVarChar, 255) { Value = LockResource });
|
||||
await cmd.ExecuteNonQueryAsync(CancellationToken.None).ConfigureAwait(false);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
// A pooled connection keeps its server session (sp_reset_connection does not release a Session-owned
|
||||
// applock); clearing the pool makes the dispose physically close it, which releases the lock.
|
||||
_log.LogWarning("sp_releaseapplock failed ({Exception}); discarding the connection pool so the session " +
|
||||
"holding the retention applock is closed rather than pooled.", ex.GetType().Name);
|
||||
ClearPool(conn);
|
||||
}
|
||||
}
|
||||
}
|
||||
+9
-5
@@ -14,12 +14,16 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer;
|
||||
/// <see cref="BootstrapFailureCache.DefaultDuration"/> retries (<see cref="BootstrapFailureCache"/>). Within that window the last failure is re-thrown without touching
|
||||
/// SQL, so a missing schema (or a SQL outage) does not serialise every write, read and alarm event behind one
|
||||
/// semaphore and one round-trip each.
|
||||
/// With <c>AutoCreateSchema=false</c> it only verifies <c>MAX(Version) >= 1</c>.
|
||||
/// With <c>AutoCreateSchema=false</c> it only verifies <c>MAX(Version) >= </c><see cref="RequiredVersion"/>.
|
||||
/// </summary>
|
||||
internal sealed partial class SqlServerSchemaBootstrapper
|
||||
{
|
||||
/// <summary>The schema version this build requires.</summary>
|
||||
public const int RequiredVersion = 1;
|
||||
/// <summary>
|
||||
/// The schema version this build requires: 2 = <c>002_sample_modified.sql</c> (Part 11 modified-value
|
||||
/// history, Gitea #668). An existing v1 schema is upgraded in place on the next bootstrap when
|
||||
/// <c>AutoCreateSchema</c> is on; otherwise the script must be applied by hand.
|
||||
/// </summary>
|
||||
public const int RequiredVersion = 2;
|
||||
|
||||
private const string LockResource = "OtOpcUaHistorianSchema";
|
||||
private const int LockTimeoutMs = 30_000;
|
||||
@@ -48,7 +52,7 @@ internal sealed partial class SqlServerSchemaBootstrapper
|
||||
/// <summary>Ensures the schema is ready; no-op after the first success.</summary>
|
||||
/// <param name="ct">Cancellation.</param>
|
||||
/// <returns>A task completing when the schema is usable.</returns>
|
||||
/// <exception cref="HistorianSchemaMissingException">AutoCreateSchema is false and version 1 is absent.</exception>
|
||||
/// <exception cref="HistorianSchemaMissingException">AutoCreateSchema is false and <see cref="RequiredVersion"/> is absent.</exception>
|
||||
public async Task EnsureAsync(CancellationToken ct)
|
||||
{
|
||||
if (_ready) return;
|
||||
@@ -94,7 +98,7 @@ internal sealed partial class SqlServerSchemaBootstrapper
|
||||
if (v < RequiredVersion)
|
||||
throw new HistorianSchemaMissingException(
|
||||
$"Historian schema [{_conn.Schema}] is missing version {RequiredVersion} (found {v}) and AutoCreateSchema is false. " +
|
||||
"Apply docs/historian/sqlserver/001_initial.sql or enable AutoCreateSchema.");
|
||||
"Apply the docs/historian/sqlserver/NNN_*.sql scripts above version " + v + " in order, or enable AutoCreateSchema.");
|
||||
}
|
||||
|
||||
private async Task ApplyAsync(SqlConnection conn, CancellationToken ct)
|
||||
|
||||
+4
-1
@@ -101,7 +101,10 @@ public sealed record OpcUaClientDriverOptions
|
||||
/// <summary>Client-side keep-alive interval.</summary>
|
||||
public TimeSpan KeepAliveInterval { get; init; } = TimeSpan.FromSeconds(5);
|
||||
|
||||
/// <summary>Initial reconnect delay after a session drop.</summary>
|
||||
/// <summary>
|
||||
/// Retained for config compatibility only — no longer used (Gitea #670). The driver has no in-driver reconnect
|
||||
/// handler; a lost session is recovered by the host's comms-loss watchdog re-initialising the driver.
|
||||
/// </summary>
|
||||
public TimeSpan ReconnectPeriod { get; init; } = TimeSpan.FromSeconds(5);
|
||||
|
||||
/// <summary>
|
||||
|
||||
@@ -22,12 +22,10 @@ internal sealed class OpcUaClientDiagnostics
|
||||
public const string PublishError = "publish_error";
|
||||
/// <summary>A keep-alive came back Bad.</summary>
|
||||
public const string KeepAliveMiss = "keepalive_miss";
|
||||
/// <summary>The reconnect handler was armed.</summary>
|
||||
public const string ReconnectAttempt = "reconnect_attempt";
|
||||
/// <summary>A reconnect landed a new session.</summary>
|
||||
public const string ReconnectSuccess = "reconnect_success";
|
||||
/// <summary>A reconnect handler exhausted its retry window.</summary>
|
||||
public const string ReconnectExhausted = "reconnect_exhausted";
|
||||
/// <summary>A keep-alive came back Good after a bad one on the same session (an in-place recovery, #670).</summary>
|
||||
public const string KeepAliveRecovered = "keepalive_recovered";
|
||||
/// <summary>A late replayed notification (source time at or below the ref's high-water mark) was dropped inside the post-recovery replay window (#670 F2).</summary>
|
||||
public const string StaleNotificationDropped = "stale_notification_dropped";
|
||||
|
||||
private readonly string _driverInstanceId;
|
||||
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,16 +1,101 @@
|
||||
using Microsoft.Extensions.Logging;
|
||||
using Opc.Ua;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient;
|
||||
|
||||
/// <summary>
|
||||
/// The one <see cref="ITelemetryContext" /> the driver hands the SDK's non-obsolete (telemetry-taking)
|
||||
/// overloads. An empty builder keeps SDK internals off any logging provider (mirrors Client.Shared's
|
||||
/// <c>ClientTelemetry</c>).
|
||||
/// The <see cref="ITelemetryContext" /> the driver hands the SDK's non-obsolete (telemetry-taking)
|
||||
/// overloads. <see cref="Context"/> has no logging provider (static helpers and tests);
|
||||
/// <see cref="ForLogger"/> routes the SDK's own Warning-and-above output (keep-alive, publish and
|
||||
/// channel errors) to a driver instance's logger — without it every SDK-side session error was
|
||||
/// invisible (Gitea #670).
|
||||
/// </summary>
|
||||
internal static class SdkTelemetry
|
||||
{
|
||||
private static readonly Lazy<ITelemetryContext> Instance = new(() => DefaultTelemetry.Create(_ => { }));
|
||||
|
||||
/// <summary>Gets the shared telemetry context.</summary>
|
||||
/// <summary>Gets the shared, provider-less telemetry context.</summary>
|
||||
public static ITelemetryContext Context => Instance.Value;
|
||||
|
||||
/// <summary>
|
||||
/// Creates a telemetry context whose SDK log output at Warning and above goes to <paramref name="logger"/>,
|
||||
/// rate-limited (a dead upstream makes the SDK warn on every keep-alive and publish for as long as the
|
||||
/// outage lasts). The context owns a <see cref="ILoggerFactory"/>: release it with <see cref="Release"/>
|
||||
/// when the driver is disposed — <see cref="DefaultTelemetry"/> is not itself disposable.
|
||||
/// </summary>
|
||||
/// <param name="logger">The driver instance's logger.</param>
|
||||
/// <returns>The telemetry context.</returns>
|
||||
public static ITelemetryContext ForLogger(ILogger logger) =>
|
||||
DefaultTelemetry.Create(builder => builder
|
||||
.SetMinimumLevel(LogLevel.Warning)
|
||||
.AddProvider(new ForwardingLoggerProvider(new RateLimitedLogger(logger, MaxEntriesPerWindow, Window))));
|
||||
|
||||
/// <summary>Disposes the logger factory a <see cref="ForLogger"/> context owns. Never the shared one.</summary>
|
||||
/// <param name="context">The context to release.</param>
|
||||
public static void Release(ITelemetryContext context)
|
||||
{
|
||||
if (ReferenceEquals(context, Context)) return;
|
||||
try { (context as TelemetryContextBase)?.LoggerFactory?.Dispose(); }
|
||||
catch { /* best-effort */ }
|
||||
}
|
||||
|
||||
/// <summary>SDK log entries forwarded per <see cref="Window"/> before the rest are dropped and counted.</summary>
|
||||
internal const int MaxEntriesPerWindow = 20;
|
||||
|
||||
/// <summary>The rate-limit window.</summary>
|
||||
internal static readonly TimeSpan Window = TimeSpan.FromMinutes(1);
|
||||
|
||||
/// <summary>Hands every SDK logger category the one (rate-limited) driver logger.</summary>
|
||||
private sealed class ForwardingLoggerProvider(ILogger target) : ILoggerProvider
|
||||
{
|
||||
public ILogger CreateLogger(string categoryName) => target;
|
||||
|
||||
public void Dispose() { }
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Forwards at most <c>max</c> entries per window; the first entry of the next window carries a note of how
|
||||
/// many were suppressed.
|
||||
/// </summary>
|
||||
internal sealed class RateLimitedLogger(ILogger target, int max, TimeSpan window) : ILogger
|
||||
{
|
||||
private readonly object _lock = new();
|
||||
private DateTime _windowStart = DateTime.MinValue;
|
||||
private int _count;
|
||||
private int _suppressed;
|
||||
|
||||
public IDisposable? BeginScope<TState>(TState state) where TState : notnull => target.BeginScope(state);
|
||||
|
||||
public bool IsEnabled(LogLevel logLevel) => target.IsEnabled(logLevel);
|
||||
|
||||
public void Log<TState>(LogLevel logLevel, EventId eventId, TState state, Exception? exception,
|
||||
Func<TState, Exception?, string> formatter)
|
||||
{
|
||||
int suppressedNote;
|
||||
lock (_lock)
|
||||
{
|
||||
var now = DateTime.UtcNow;
|
||||
if (now - _windowStart >= window)
|
||||
{
|
||||
suppressedNote = _suppressed;
|
||||
_windowStart = now;
|
||||
_count = 0;
|
||||
_suppressed = 0;
|
||||
}
|
||||
else suppressedNote = 0;
|
||||
|
||||
if (_count >= max)
|
||||
{
|
||||
_suppressed++;
|
||||
return;
|
||||
}
|
||||
_count++;
|
||||
}
|
||||
|
||||
if (suppressedNote > 0)
|
||||
target.LogWarning("OPC UA SDK: {Suppressed} further log entries were suppressed in the last {Window}.",
|
||||
suppressedNote, window);
|
||||
target.Log(logLevel, eventId, state, exception, formatter);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
+2
@@ -31,6 +31,8 @@
|
||||
<InternalsVisibleTo Include="ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient.Tests"/>
|
||||
<!-- Gitea #656: the native-alarm re-statement end-to-end tests drive this driver into the real DriverHostActor. -->
|
||||
<InternalsVisibleTo Include="ZB.MOM.WW.OtOpcUa.Runtime.Tests"/>
|
||||
<!-- Gitea #670 review M1: the wire test deletes the driver's subscription server-side through its own session. -->
|
||||
<InternalsVisibleTo Include="ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests"/>
|
||||
</ItemGroup>
|
||||
|
||||
</Project>
|
||||
|
||||
@@ -41,8 +41,8 @@ public static class HistorianBackendRegistry
|
||||
new Dictionary<string, Func<ServerHistorianOptions, IConfiguration, IHistorianBackendProvider>>(StringComparer.OrdinalIgnoreCase)
|
||||
{
|
||||
[ServerHistorianOptions.GatewayBackendName] = (o, _) => new GatewayHistorianProvider(o),
|
||||
[SqlServerBackendName] = (_, c) => new SqlServerHistorianProvider(BindSqlServer(c)),
|
||||
[QuestDbBackendName] = (_, c) => new QuestDbHistorianProvider(BindQuestDb(c)),
|
||||
[SqlServerBackendName] = (o, c) => new SqlServerHistorianProvider(BindSqlServer(c), o.MaxTimeWeightedRawRows),
|
||||
[QuestDbBackendName] = (o, c) => new QuestDbHistorianProvider(BindQuestDb(c), o.MaxTimeWeightedRawRows),
|
||||
};
|
||||
|
||||
/// <summary>The registered backend names, for validation messages.</summary>
|
||||
|
||||
@@ -326,6 +326,14 @@ if (hasDriver)
|
||||
builder.Services.AddSingleton<IHistorianValueWriter>(sp => sp.GetRequiredService<IHistorianBackendProvider>().CreateValueWriter(sp));
|
||||
}
|
||||
|
||||
// Historian retention (#667). Only a backend that manages retention in-process returns a purger
|
||||
// (SqlServer with RetentionDays set); otherwise the service is inert. Primary-gated via IHistorizationRoleView.
|
||||
if (serverHistorianOptions.Enabled)
|
||||
builder.Services.AddHostedService(sp => new HistorianRetentionService(
|
||||
sp.GetRequiredService<IHistorianBackendProvider>().CreateRetention(sp),
|
||||
sp.GetService<ZB.MOM.WW.OtOpcUa.Runtime.Redundancy.IHistorizationRoleView>(),
|
||||
sp.GetService<ILogger<HistorianRetentionService>>()));
|
||||
|
||||
// Bind every cross-platform driver factory before AddAkka resolves IDriverFactory — replaces
|
||||
// the F7-default NullDriverFactory with a real DriverFactoryRegistryAdapter so DriverHostActor
|
||||
// can materialise real IDriver instances on deploy.
|
||||
|
||||
@@ -55,7 +55,9 @@
|
||||
"MaxWriteAttempts": 5,
|
||||
"_MaxWriteAttemptsComment": "Gitea #538: consecutive failed drain passes one tag may accumulate — counted only across passes where OTHER tags wrote successfully — before its queued entries are abandoned so the FIFO head can advance.",
|
||||
"MinBackoffSeconds": 1,
|
||||
"MaxBackoffSeconds": 30
|
||||
"MaxBackoffSeconds": 30,
|
||||
"CaptureWhilePrimaryUnreachable": true,
|
||||
"_CaptureWhilePrimaryUnreachableComment": "Gitea #672: a standby stood down for its mesh Primary resumes capture as soon as that Primary is reported UNREACHABLE (not after auto-down) — shrinks a crash-failover capture gap to the failure-detector window, at the cost of short duplicate windows. false = pre-#672."
|
||||
},
|
||||
"AlarmHistorian": {
|
||||
"_comment": "Durable store-and-forward alarm sink. Buffers into the node's consolidated LocalDb (see the LocalDb section) so the queue replicates to the redundant pair peer, and drains alarm events to the ServerHistorian gateway's SendEvent path from whichever node holds the Primary role. The downstream connection (endpoint/key/TLS) is sourced from the ServerHistorian section.",
|
||||
|
||||
@@ -31,12 +31,43 @@ namespace ZB.MOM.WW.OtOpcUa.OpcUaServer;
|
||||
/// prior pages and must be dropped from the head of the next page (tie de-dup).
|
||||
/// </param>
|
||||
/// <param name="NumValuesPerNode">The client's per-page cap; re-applied to every resumed page.</param>
|
||||
/// <param name="Kind">
|
||||
/// Which read issued the cursor (Gitea #669 / #668). A cursor is only ever resumed by the read kind that issued it, so a
|
||||
/// raw point presented to an events or modified read (or any other mismatch) is a <c>BadContinuationPointInvalid</c>
|
||||
/// miss. <see cref="HistoryCursorKind.Events"/>: <paramref name="Tagname"/> carries the registered historian
|
||||
/// event-source name and the boundary fields are over <c>EventTimeUtc</c>.
|
||||
/// </param>
|
||||
/// <param name="LastModifiedId">
|
||||
/// <see cref="HistoryCursorKind.Modified"/> only: the <c>ModifiedId</c> of the last returned modification. With
|
||||
/// <paramref name="NextStartUtc"/> (its source time) it is the exclusive resume key; <paramref name="BoundarySkipCount"/>
|
||||
/// is unused because the key is unique.
|
||||
/// </param>
|
||||
/// <param name="WindowStartUtc">
|
||||
/// <see cref="HistoryCursorKind.Modified"/> only: the normalised window start, so a resumed page keeps the original
|
||||
/// window and direction (the modified read resumes by key, not by moving the window start).
|
||||
/// </param>
|
||||
internal sealed record HistoryContinuationState(
|
||||
string Tagname,
|
||||
DateTime NextStartUtc,
|
||||
DateTime EndUtc,
|
||||
int BoundarySkipCount,
|
||||
uint NumValuesPerNode);
|
||||
uint NumValuesPerNode,
|
||||
HistoryCursorKind Kind = HistoryCursorKind.Raw,
|
||||
long LastModifiedId = 0,
|
||||
DateTime WindowStartUtc = default);
|
||||
|
||||
/// <summary>The HistoryRead kind that issued a <see cref="HistoryContinuationState"/>.</summary>
|
||||
internal enum HistoryCursorKind
|
||||
{
|
||||
/// <summary>HistoryReadRaw (values).</summary>
|
||||
Raw,
|
||||
|
||||
/// <summary>HistoryReadEvents (#669).</summary>
|
||||
Events,
|
||||
|
||||
/// <summary>HistoryReadRaw with <c>IsReadModified = true</c> — the Part 11 modification trail (#668).</summary>
|
||||
Modified,
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Pure server-side continuation-point paging decisions for the count-capped variable-history arms
|
||||
|
||||
@@ -4977,8 +4977,12 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
|
||||
{
|
||||
if (details.IsReadModified)
|
||||
{
|
||||
// We never serve modified-value history; mark this node unsupported and move on.
|
||||
errors[handle.Index] = StatusCodes.BadHistoryOperationUnsupported;
|
||||
// Part 11 modified-value history (#668): the prior values every HistoryUpdate recorded. Gated by the
|
||||
// same HistoryRead authorization as a raw read (above); a backend with no trail ⇒ unsupported.
|
||||
work.Add(() => ServeModifiedPagedAsync(
|
||||
handle, session, nodesToRead, results, errors,
|
||||
details.StartTime, details.EndTime, details.NumValuesPerNode,
|
||||
ct));
|
||||
continue;
|
||||
}
|
||||
|
||||
@@ -5086,6 +5090,7 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
|
||||
{
|
||||
// Snapshot the select clauses once — the same filter projects every node's events.
|
||||
var selectClauses = details.Filter?.SelectClauses ?? new SimpleAttributeOperandCollection();
|
||||
var session = context.OperationContext?.Session;
|
||||
|
||||
nodesToProcess = AuthorizeHistoryRead(context, nodesToProcess, results, errors);
|
||||
WithHistoryReadBatchLimiter(nodesToProcess, results, errors, () =>
|
||||
@@ -5108,7 +5113,7 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
|
||||
}
|
||||
|
||||
var sn = sourceName;
|
||||
work.Add(() => ServeEventsAsync(handle, sn, details, selectClauses, results, errors, ct));
|
||||
work.Add(() => ServeEventsAsync(handle, sn, details, selectClauses, nodesToRead, session, results, errors, ct));
|
||||
}
|
||||
|
||||
RunBounded(work, HistoryReadBatchParallelism);
|
||||
@@ -5134,32 +5139,130 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
|
||||
string? sourceName,
|
||||
ReadEventDetails details,
|
||||
SimpleAttributeOperandCollection selectClauses,
|
||||
IList<HistoryReadValueId> nodesToRead,
|
||||
ISession? session,
|
||||
IList<SdkHistoryReadResult> results,
|
||||
IList<ServiceResult> errors,
|
||||
CancellationToken ct)
|
||||
{
|
||||
var inboundCp = nodesToRead[handle.Index].ContinuationPoint;
|
||||
var resumed = inboundCp is { Length: > 0 };
|
||||
try
|
||||
{
|
||||
// NOT under the node-manager Lock — awaiting the async source is safe here.
|
||||
var sourceResult = await _historianDataSource.ReadEventsAsync(
|
||||
sourceName,
|
||||
details.StartTime,
|
||||
details.EndTime,
|
||||
// NumValuesPerNode==0 means "no limit — return ALL values" per OPC UA
|
||||
// Part 4/11, so translate it to UNBOUNDED (int.MaxValue) here. Passing the int<=0
|
||||
// backend-default-cap sentinel instead would silently truncate a "give me everything"
|
||||
// events read at the backend default. A positive cap passes through (clamped).
|
||||
EventMaxEvents(details.NumValuesPerNode),
|
||||
ct).ConfigureAwait(false);
|
||||
var startUtc = details.StartTime;
|
||||
var endUtc = details.EndTime;
|
||||
var numValuesPerNode = details.NumValuesPerNode;
|
||||
var boundarySkip = 0;
|
||||
|
||||
var historyEvent = ProjectEvents(sourceResult.Events, selectClauses);
|
||||
if (resumed)
|
||||
{
|
||||
// Single-use take. A miss (unknown / evicted / malformed / a RAW cursor) ⇒ BadContinuationPointInvalid
|
||||
// for THIS node only.
|
||||
var state = _historyContinuationStore.TryTake(session, inboundCp!);
|
||||
// A point is bound to the node that issued it: another node's source name is a miss too.
|
||||
if (state is null || state.Kind != HistoryCursorKind.Events || state.Tagname != (sourceName ?? string.Empty))
|
||||
{
|
||||
errors[handle.Index] = StatusCodes.BadContinuationPointInvalid;
|
||||
results[handle.Index] = new SdkHistoryReadResult { StatusCode = StatusCodes.BadContinuationPointInvalid };
|
||||
return;
|
||||
}
|
||||
|
||||
sourceName = state.Tagname;
|
||||
startUtc = state.NextStartUtc;
|
||||
endUtc = state.EndUtc;
|
||||
boundarySkip = state.BoundarySkipCount;
|
||||
numValuesPerNode = state.NumValuesPerNode;
|
||||
}
|
||||
|
||||
// Part 11: an unset EndTime (MinValue) with a StartTime means "page forward from StartTime". Normalise it
|
||||
// to an open upper bound, or the cursor would store MinValue and a backend would swap the reversed
|
||||
// resume window. A resumed read is always forward (the cursor is only ever stored for a forward read),
|
||||
// so its direction comes from the cursor, not from the resume request's details.
|
||||
if (!resumed && endUtc == DateTime.MinValue && startUtc != DateTime.MinValue)
|
||||
endUtc = new DateTime(DateTime.MaxValue.Ticks, DateTimeKind.Utc); // Utc kind: a proto Timestamp conversion rejects Unspecified
|
||||
// Reverse event reads (start > end) are not paged: the cursor below is forward-only.
|
||||
var forward = resumed || startUtc <= endUtc;
|
||||
|
||||
// NOT under the node-manager Lock — awaiting the async source is safe here.
|
||||
// NumValuesPerNode==0 means "no limit — return ALL values" per OPC UA Part 4/11, so it is translated
|
||||
// to UNBOUNDED (EventMaxEvents) rather than the backend's <=0 default-cap sentinel.
|
||||
var sourceResult = await _historianDataSource.ReadEventsAsync(
|
||||
sourceName, startUtc, endUtc, EventMaxEvents(numValuesPerNode), ct).ConfigureAwait(false);
|
||||
|
||||
// Truncation is the SOURCE's word (non-null continuation point) — never inferred from the count, so a
|
||||
// page that happens to be exactly cap-sized but complete never emits a point.
|
||||
var truncated = sourceResult.ContinuationPoint is not null && numValuesPerNode > 0 && forward;
|
||||
|
||||
IReadOnlyList<HistoricalEvent> events = sourceResult.Events;
|
||||
if (resumed && boundarySkip > 0)
|
||||
{
|
||||
// Drop the events at the boundary timestamp already emitted on the prior page.
|
||||
var i = 0;
|
||||
while (i < events.Count && i < boundarySkip && events[i].EventTimeUtc == startUtc) i++;
|
||||
if (i > 0) events = events.Skip(i).ToList();
|
||||
}
|
||||
|
||||
// The cursor to store, if any. Saved only AFTER the page is projected, so a projection throw can
|
||||
// never orphan a stored cursor.
|
||||
HistoryContinuationState? pendingCursor = null;
|
||||
|
||||
if (truncated && resumed
|
||||
&& (events.Count == 0 || events[^1].EventTimeUtc == startUtc))
|
||||
{
|
||||
// Oversized tie cluster: more events share startUtc than the page cap, so (time, skip) alone
|
||||
// cannot advance. Over-fetch the whole cluster over the half-open [T, T+resolution) window
|
||||
// with an explicit bounded cap and page WITHIN it (same scheme as raw).
|
||||
var resolution = Math.Max(1L, _historianDataSource.TimestampResolutionTicks);
|
||||
var overfetchCap = (int)Math.Min(int.MaxValue - 1L, (long)Math.Max(1, MaxTieClusterOverfetch) + 1);
|
||||
var cluster = (await _historianDataSource.ReadEventsAsync(
|
||||
sourceName, startUtc, startUtc.AddTicks(resolution), overfetchCap, ct)
|
||||
.ConfigureAwait(false)).Events;
|
||||
|
||||
if (cluster.Count > MaxTieClusterOverfetch)
|
||||
{
|
||||
_logger.LogError(
|
||||
"OtOpcUaNodeManager: HistoryReadEvents tie cluster at {StartUtc:O} for source '{Source}' has {Count} events, " +
|
||||
"exceeding MaxTieClusterOverfetch={Max}; cannot page within it. Increase MaxTieClusterOverfetch.",
|
||||
startUtc, sourceName, cluster.Count, MaxTieClusterOverfetch);
|
||||
errors[handle.Index] = StatusCodes.BadHistoryOperationUnsupported;
|
||||
results[handle.Index] = new SdkHistoryReadResult { StatusCode = StatusCodes.BadHistoryOperationUnsupported };
|
||||
return;
|
||||
}
|
||||
|
||||
HistoryPaging.SliceTieCluster(
|
||||
cluster.Count, boundarySkip, numValuesPerNode, startUtc, endUtc,
|
||||
out var sliceStart, out var sliceCount, out var nextStartUtc, out var nextSkip,
|
||||
reverse: false, resolutionTicks: resolution);
|
||||
|
||||
var slice = new List<HistoricalEvent>(sliceCount);
|
||||
for (var i = sliceStart; i < sliceStart + sliceCount; i++) slice.Add(cluster[i]);
|
||||
events = slice;
|
||||
|
||||
if (nextStartUtc is { } resumeAt)
|
||||
{
|
||||
pendingCursor = new HistoryContinuationState(
|
||||
sourceName ?? string.Empty, resumeAt, endUtc, nextSkip, numValuesPerNode, HistoryCursorKind.Events);
|
||||
}
|
||||
}
|
||||
else if (truncated && events.Count > 0)
|
||||
{
|
||||
// Cursor from the last event: resume at its time INCLUSIVE, skipping the trailing ties emitted.
|
||||
var boundary = events[^1].EventTimeUtc;
|
||||
var skip = 0;
|
||||
for (var i = events.Count - 1; i >= 0 && events[i].EventTimeUtc == boundary; i--) skip++;
|
||||
pendingCursor = new HistoryContinuationState(
|
||||
sourceName ?? string.Empty, boundary, endUtc, skip, numValuesPerNode, HistoryCursorKind.Events);
|
||||
}
|
||||
|
||||
var historyEvent = ProjectEvents(events, selectClauses);
|
||||
var outboundCp = pendingCursor is null ? null : _historyContinuationStore.Save(session, pendingCursor);
|
||||
results[handle.Index] = new SdkHistoryReadResult
|
||||
{
|
||||
// No events ⇒ GoodNoData (the notifier is historized, the window just held no events).
|
||||
StatusCode = sourceResult.Events.Count == 0 ? StatusCodes.GoodNoData : StatusCodes.Good,
|
||||
StatusCode = events.Count == 0 ? StatusCodes.GoodNoData : StatusCodes.Good,
|
||||
HistoryData = new ExtensionObject(historyEvent),
|
||||
// We never issue continuation points — every read returns the full window in one shot.
|
||||
ContinuationPoint = null,
|
||||
// Null when the page was complete, or when no session is available to hold the cursor.
|
||||
ContinuationPoint = outboundCp,
|
||||
};
|
||||
errors[handle.Index] = ServiceResult.Good;
|
||||
}
|
||||
@@ -5275,7 +5378,7 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
|
||||
}
|
||||
|
||||
var items = (IList<DataValue>?)details.UpdateValues ?? Array.Empty<DataValue>();
|
||||
return (updater, tagname, ct) => UpdateDataAsync(updater, tagname, mode, items, ct);
|
||||
return (updater, tagname, who, ct) => UpdateDataAsync(updater, tagname, mode, items, who, ct);
|
||||
});
|
||||
}
|
||||
|
||||
@@ -5291,17 +5394,18 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
|
||||
ServeHistoryUpdate(context, nodesToProcess, results, errors, handle =>
|
||||
{
|
||||
var details = nodesToUpdate[handle.Index];
|
||||
// We never keep modified-value history, so there is none to delete.
|
||||
// Deleting the modified-value trail itself (Part 11 IsDeleteModified) is not supported: the trail is the
|
||||
// audit record of every change and is aged out only by retention (#668).
|
||||
if (details.IsDeleteModified) return null;
|
||||
var start = details.StartTime;
|
||||
var end = details.EndTime;
|
||||
// Part 11 requires both bounds of a raw delete; DateTime.MinValue is the wire's "unspecified". Refuse
|
||||
// it here rather than let a backend read MinValue as "from the beginning of time".
|
||||
if (start == DateTime.MinValue || end == DateTime.MinValue)
|
||||
return (_, _, _) => Task.FromResult(new HistoryUpdateResult { StatusCode = StatusCodes.BadInvalidArgument });
|
||||
return async (updater, tagname, ct) =>
|
||||
return (_, _, _, _) => Task.FromResult(new HistoryUpdateResult { StatusCode = StatusCodes.BadInvalidArgument });
|
||||
return async (updater, tagname, who, ct) =>
|
||||
{
|
||||
var code = await updater.DeleteRawAsync(tagname, start, end, ct).ConfigureAwait(false);
|
||||
var code = await updater.DeleteRawAsync(tagname, start, end, who, ct).ConfigureAwait(false);
|
||||
return new HistoryUpdateResult { StatusCode = code };
|
||||
};
|
||||
});
|
||||
@@ -5319,11 +5423,11 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
|
||||
ServeHistoryUpdate(context, nodesToProcess, results, errors, handle =>
|
||||
{
|
||||
var times = nodesToUpdate[handle.Index].ReqTimes?.ToList() ?? new List<DateTime>();
|
||||
return async (updater, tagname, ct) =>
|
||||
return async (updater, tagname, who, ct) =>
|
||||
{
|
||||
var codes = times.Count == 0
|
||||
? Array.Empty<uint>()
|
||||
: await updater.DeleteAtTimeAsync(tagname, times, ct).ConfigureAwait(false);
|
||||
: await updater.DeleteAtTimeAsync(tagname, times, who, ct).ConfigureAwait(false);
|
||||
if (codes.Count != times.Count)
|
||||
throw new InvalidOperationException(
|
||||
$"historian updater returned {codes.Count} result(s) for {times.Count} time(s)");
|
||||
@@ -5346,11 +5450,12 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
|
||||
/// <param name="tagname">The node's historian tagname.</param>
|
||||
/// <param name="mode">The update mode.</param>
|
||||
/// <param name="items">The request's values, in order.</param>
|
||||
/// <param name="who">The caller, recorded in the modification trail.</param>
|
||||
/// <param name="ct">The request deadline.</param>
|
||||
/// <returns>The node's result, carrying one operation result per item.</returns>
|
||||
private static async Task<HistoryUpdateResult> UpdateDataAsync(
|
||||
IHistorianDataUpdater updater, string tagname, HistoryUpdateMode mode, IList<DataValue> items,
|
||||
CancellationToken ct)
|
||||
HistoryUpdateContext who, CancellationToken ct)
|
||||
{
|
||||
var codes = new uint[items.Count];
|
||||
var toSend = new List<HistorizationValue>(items.Count);
|
||||
@@ -5379,7 +5484,7 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
|
||||
|
||||
if (toSend.Count > 0)
|
||||
{
|
||||
var backend = await updater.UpdateDataAsync(tagname, mode, toSend, ct).ConfigureAwait(false);
|
||||
var backend = await updater.UpdateDataAsync(tagname, mode, toSend, who, ct).ConfigureAwait(false);
|
||||
if (backend.Count != toSend.Count)
|
||||
throw new InvalidOperationException(
|
||||
$"historian updater returned {backend.Count} result(s) for {toSend.Count} value(s)");
|
||||
@@ -5411,11 +5516,14 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
|
||||
List<NodeHandle> nodesToProcess,
|
||||
IList<HistoryUpdateResult> results,
|
||||
IList<ServiceResult> errors,
|
||||
Func<NodeHandle, Func<IHistorianDataUpdater, string, CancellationToken, Task<HistoryUpdateResult>>?> plan)
|
||||
Func<NodeHandle, Func<IHistorianDataUpdater, string, HistoryUpdateContext, CancellationToken, Task<HistoryUpdateResult>>?> plan)
|
||||
{
|
||||
nodesToProcess = AuthorizeHistoryUpdate(context, nodesToProcess, results, errors);
|
||||
if (nodesToProcess.Count == 0) return;
|
||||
|
||||
// Who is rewriting history — recorded on every modification-trail row (Part 11 ModificationInfo.UserName, #668).
|
||||
var who = HistoryUpdateCallerOf(context);
|
||||
|
||||
WithHistoryReadBatchLimiter(nodesToProcess, results, errors, () =>
|
||||
{
|
||||
using var deadline = CreateDeadline();
|
||||
@@ -5434,13 +5542,30 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
|
||||
|
||||
var h = handle;
|
||||
var tag = tagname;
|
||||
work.Add(() => ServeUpdateNodeAsync(h, results, errors, token => operation(updater!, tag, token), ct));
|
||||
work.Add(() => ServeUpdateNodeAsync(h, results, errors, token => operation(updater!, tag, who, token), ct));
|
||||
}
|
||||
|
||||
RunBounded(work, HistoryReadBatchParallelism);
|
||||
});
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The <see cref="HistoryUpdateContext"/> of the session behind <paramref name="context"/>: its API-key principal
|
||||
/// (or the reserved <c>anonymous</c> principal) and key id. Only an identity that already passed
|
||||
/// <see cref="AuthorizeHistoryUpdate"/> reaches a backend, so the anonymous fallback for a missing identity is
|
||||
/// defence in depth, not a path.
|
||||
/// </summary>
|
||||
/// <param name="context">The SDK context carrying the session identity.</param>
|
||||
/// <returns>The caller recorded in the modification trail.</returns>
|
||||
internal static HistoryUpdateContext HistoryUpdateCallerOf(ServerSystemContext context)
|
||||
{
|
||||
var identity = (context as ISessionOperationContext)?.UserIdentity as RoleCarryingUserIdentity
|
||||
?? context.OperationContext?.UserIdentity as RoleCarryingUserIdentity;
|
||||
return identity is null
|
||||
? new HistoryUpdateContext(OpcUaPrincipals.Anonymous, null)
|
||||
: new HistoryUpdateContext(identity.Principal, identity.ApiKeyId);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Runs one node's HistoryUpdate operation and fills its slots. Never throws (the
|
||||
/// <see cref="RunBounded"/> contract): a deadline cancellation is <c>BadTimeout</c>, any backend failure
|
||||
@@ -5779,6 +5904,14 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
|
||||
errors[handle.Index] = StatusCodes.BadAggregateNotSupported;
|
||||
results[handle.Index] = new SdkHistoryReadResult { StatusCode = StatusCodes.BadAggregateNotSupported };
|
||||
}
|
||||
catch (HistoryReadLimitExceededException ex)
|
||||
{
|
||||
// The read would exceed a backend size limit (#671: the time-weighted raw-row cap) — refuse THIS node
|
||||
// explicitly instead of a generic fault or a deadline-elapsed BadTimeout.
|
||||
_logger.LogWarning("OtOpcUaNodeManager: HistoryRead limit exceeded for node {NodeId}: {Message}", handle.NodeId, ex.Message);
|
||||
errors[handle.Index] = StatusCodes.BadTooManyOperations;
|
||||
results[handle.Index] = new SdkHistoryReadResult { StatusCode = StatusCodes.BadTooManyOperations };
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
// One node's backend failure (throw / timeout) must not throw out of the batch — surface a
|
||||
@@ -5843,7 +5976,8 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
|
||||
// Resume read: take the stored cursor. A miss (unknown / evicted / malformed point) ⇒
|
||||
// BadContinuationPointInvalid for THIS node.
|
||||
var state = _historyContinuationStore.TryTake(session, inboundCp);
|
||||
if (state is null)
|
||||
if (state is null || state.Kind != HistoryCursorKind.Raw
|
||||
|| !TryGetHistorizedTagname(handle.NodeId, out var boundTag) || boundTag != state.Tagname)
|
||||
{
|
||||
errors[handle.Index] = StatusCodes.BadContinuationPointInvalid;
|
||||
results[handle.Index] = new SdkHistoryReadResult { StatusCode = StatusCodes.BadContinuationPointInvalid };
|
||||
@@ -5867,6 +6001,17 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
|
||||
|
||||
tagname = resolved;
|
||||
startUtc = startTimeUtc;
|
||||
|
||||
// Part 11: with NumValuesPerNode > 0 one bound may be left unset. Only a StartTime ⇒ page FORWARD from
|
||||
// it to an open upper bound; only an EndTime ⇒ page BACKWARD from it (a reverse read, start > end).
|
||||
// Without this an unset EndTime (MinValue) looked like a reverse read to a lower bound of "now".
|
||||
if (numValuesPerNode > 0)
|
||||
{
|
||||
if (endUtc == DateTime.MinValue && startUtc != DateTime.MinValue)
|
||||
endUtc = new DateTime(DateTime.MaxValue.Ticks, DateTimeKind.Utc);
|
||||
else if (startUtc == DateTime.MinValue && endUtc != DateTime.MinValue)
|
||||
(startUtc, endUtc) = (endUtc, new DateTime(0, DateTimeKind.Utc));
|
||||
}
|
||||
}
|
||||
|
||||
// HistoryRead is NOT under the node-manager Lock — awaiting the async source is safe.
|
||||
@@ -5997,6 +6142,138 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Serve one historized variable for a HistoryReadRaw with <c>IsReadModified = true</c> (Part 11 modified-value
|
||||
/// history, Gitea #668), with continuation-point paging. The backend orders the trail by
|
||||
/// <c>(SourceTimeUtc, ModifiedId)</c> and reports truncation itself (<see cref="HistoryModifiedReadResult.HasMore"/>),
|
||||
/// so the cursor is an exact, exclusive key — there are no ties to trim. Window normalisation matches
|
||||
/// <see cref="ServeRawPagedAsync"/>: with a cap, an unset EndTime pages forward to an open bound and an unset
|
||||
/// StartTime pages backward. A cursor is bound to the node's tagname and to the modified read kind; any other
|
||||
/// point is <c>BadContinuationPointInvalid</c>. A backend that keeps no trail (Gateway, Null) throws
|
||||
/// <see cref="NotSupportedException"/> ⇒ <c>BadHistoryOperationUnsupported</c> for this node.
|
||||
/// </summary>
|
||||
/// <param name="handle">The node handle; <c>handle.Index</c> indexes results/errors.</param>
|
||||
/// <param name="session">The session (null on the session-less in-process path ⇒ single-shot).</param>
|
||||
/// <param name="nodesToRead">The per-node read list (inbound continuation points).</param>
|
||||
/// <param name="results">The service-level results list.</param>
|
||||
/// <param name="errors">The service-level errors list.</param>
|
||||
/// <param name="startTimeUtc">Request StartTime.</param>
|
||||
/// <param name="endUtc">Request EndTime.</param>
|
||||
/// <param name="numValuesPerNode">Per-page cap; <c>0</c> = all.</param>
|
||||
/// <param name="ct">Request deadline.</param>
|
||||
private async Task ServeModifiedPagedAsync(
|
||||
NodeHandle handle,
|
||||
ISession? session,
|
||||
IList<HistoryReadValueId> nodesToRead,
|
||||
IList<SdkHistoryReadResult> results,
|
||||
IList<ServiceResult> errors,
|
||||
DateTime startTimeUtc,
|
||||
DateTime endUtc,
|
||||
uint numValuesPerNode,
|
||||
CancellationToken ct)
|
||||
{
|
||||
var inboundCp = nodesToRead[handle.Index].ContinuationPoint;
|
||||
try
|
||||
{
|
||||
if (!TryGetHistorizedTagname(handle.NodeId, out var tagname) || tagname is null)
|
||||
{
|
||||
errors[handle.Index] = StatusCodes.BadHistoryOperationUnsupported;
|
||||
results[handle.Index] = new SdkHistoryReadResult { StatusCode = StatusCodes.BadHistoryOperationUnsupported };
|
||||
return;
|
||||
}
|
||||
|
||||
var startUtc = startTimeUtc;
|
||||
HistoryModifiedCursor? resumeAfter = null;
|
||||
if (inboundCp is { Length: > 0 })
|
||||
{
|
||||
var state = _historyContinuationStore.TryTake(session, inboundCp);
|
||||
if (state is null || state.Kind != HistoryCursorKind.Modified || state.Tagname != tagname)
|
||||
{
|
||||
errors[handle.Index] = StatusCodes.BadContinuationPointInvalid;
|
||||
results[handle.Index] = new SdkHistoryReadResult { StatusCode = StatusCodes.BadContinuationPointInvalid };
|
||||
return;
|
||||
}
|
||||
|
||||
// The cursor carries the NORMALISED window, so a resumed page never re-derives its direction.
|
||||
startUtc = state.WindowStartUtc;
|
||||
endUtc = state.EndUtc;
|
||||
numValuesPerNode = state.NumValuesPerNode;
|
||||
resumeAfter = new HistoryModifiedCursor(state.NextStartUtc, state.LastModifiedId);
|
||||
}
|
||||
else if (numValuesPerNode > 0)
|
||||
{
|
||||
if (endUtc == DateTime.MinValue && startUtc != DateTime.MinValue)
|
||||
endUtc = new DateTime(DateTime.MaxValue.Ticks, DateTimeKind.Utc);
|
||||
else if (startUtc == DateTime.MinValue && endUtc != DateTime.MinValue)
|
||||
(startUtc, endUtc) = (endUtc, new DateTime(0, DateTimeKind.Utc));
|
||||
}
|
||||
|
||||
var page = await HistorianDataSource
|
||||
.ReadModifiedAsync(tagname, startUtc, endUtc, numValuesPerNode, resumeAfter, ct)
|
||||
.ConfigureAwait(false);
|
||||
|
||||
var data = ToHistoryModifiedData(page.Values);
|
||||
byte[]? outboundCp = null;
|
||||
if (page.HasMore && numValuesPerNode > 0 && page.Values.Count > 0)
|
||||
{
|
||||
var last = page.Values[^1];
|
||||
outboundCp = _historyContinuationStore.Save(session, new HistoryContinuationState(
|
||||
tagname, last.SourceTimeUtc, endUtc, 0, numValuesPerNode,
|
||||
HistoryCursorKind.Modified, last.ModifiedId, startUtc));
|
||||
}
|
||||
|
||||
results[handle.Index] = new SdkHistoryReadResult
|
||||
{
|
||||
StatusCode = page.Values.Count == 0 ? StatusCodes.GoodNoData : StatusCodes.Good,
|
||||
HistoryData = new ExtensionObject(data),
|
||||
ContinuationPoint = outboundCp,
|
||||
};
|
||||
errors[handle.Index] = ServiceResult.Good;
|
||||
}
|
||||
catch (OperationCanceledException)
|
||||
{
|
||||
errors[handle.Index] = StatusCodes.BadTimeout;
|
||||
results[handle.Index] = new SdkHistoryReadResult { StatusCode = StatusCodes.BadTimeout };
|
||||
}
|
||||
catch (NotSupportedException ex)
|
||||
{
|
||||
// The backend keeps no modification trail (Gateway / Null) — the spec status, not a server fault.
|
||||
_logger.LogDebug(ex, "OtOpcUaNodeManager: backend serves no modified-value history for node {NodeId}", handle.NodeId);
|
||||
errors[handle.Index] = StatusCodes.BadHistoryOperationUnsupported;
|
||||
results[handle.Index] = new SdkHistoryReadResult { StatusCode = StatusCodes.BadHistoryOperationUnsupported };
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogError(ex, "OtOpcUaNodeManager: HistoryRead (modified) failed for node {NodeId}", handle.NodeId);
|
||||
errors[handle.Index] = StatusCodes.BadHistoryOperationUnsupported;
|
||||
results[handle.Index] = new SdkHistoryReadResult { StatusCode = StatusCodes.BadHistoryOperationUnsupported };
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Project a modification trail into an SDK <see cref="HistoryModifiedData"/>: one prior <see cref="DataValue"/>
|
||||
/// per modification plus a parallel <see cref="ModificationInfo"/> (time, Part 11 update type, user name).
|
||||
/// </summary>
|
||||
/// <param name="values">The modifications, in the order to emit.</param>
|
||||
/// <returns>The populated <see cref="HistoryModifiedData"/>.</returns>
|
||||
internal static HistoryModifiedData ToHistoryModifiedData(IReadOnlyList<HistoryModifiedValue> values)
|
||||
{
|
||||
var dataValues = new DataValueCollection(values.Count);
|
||||
var infos = new ModificationInfoCollection(values.Count);
|
||||
foreach (var m in values)
|
||||
{
|
||||
dataValues.Add(ToSdkDataValue(m.Value));
|
||||
infos.Add(new ModificationInfo
|
||||
{
|
||||
ModificationTime = m.ModificationTimeUtc,
|
||||
UpdateType = (HistoryUpdateType)(int)m.UpdateType,
|
||||
UserName = m.UserName,
|
||||
});
|
||||
}
|
||||
|
||||
return new HistoryModifiedData { DataValues = dataValues, ModificationInfos = infos };
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
protected override void HistoryReleaseContinuationPoints(
|
||||
ServerSystemContext context,
|
||||
@@ -6063,6 +6340,8 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
|
||||
if (aggregateNodeId == ObjectIds.AggregateFunction_AnnotationCount) return HistoryAggregateType.AnnotationCount;
|
||||
if (aggregateNodeId == ObjectIds.AggregateFunction_StandardDeviationPopulation) return HistoryAggregateType.StandardDeviationPopulation;
|
||||
if (aggregateNodeId == ObjectIds.AggregateFunction_VariancePopulation) return HistoryAggregateType.VariancePopulation;
|
||||
if (aggregateNodeId == ObjectIds.AggregateFunction_TimeAverage2) return HistoryAggregateType.TimeAverage2;
|
||||
if (aggregateNodeId == ObjectIds.AggregateFunction_Total2) return HistoryAggregateType.Total2;
|
||||
return null;
|
||||
}
|
||||
|
||||
|
||||
@@ -461,6 +461,30 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
|
||||
/// </summary>
|
||||
private readonly Func<CommonsNodeId, bool>? _sharesHistorizationScopeWithLocal;
|
||||
|
||||
/// <summary>Gitea #672: whether an UNREACHABLE recording Primary still stands this node's historization
|
||||
/// capture down (<c>false</c>) or not (<c>true</c>, the default). See <see cref="ComputeHistorizationVerdict"/>.</summary>
|
||||
private readonly bool _captureWhilePrimaryUnreachable;
|
||||
|
||||
/// <summary>Gitea #672: the last redundancy snapshot's nodes, so a reachability change can re-derive the
|
||||
/// historization capture verdict without waiting for the next snapshot — which, when the recording Primary
|
||||
/// crashed, may not come until auto-down (the redundancy singleton usually lived on that Primary).</summary>
|
||||
private IReadOnlyList<NodeRedundancyState>? _lastRedundancySnapshotNodes;
|
||||
|
||||
/// <summary>Gitea #672: peers the cluster failure detector currently reports unreachable (never this node).
|
||||
/// Fed by <see cref="Akka.Cluster.ClusterEvent.ReachabilityEvent"/>; cleared for an address that comes Up again.</summary>
|
||||
private readonly HashSet<CommonsNodeId> _unreachableNodes = new();
|
||||
|
||||
/// <summary>Gitea #672: the unreachable recording Primary this node is currently capturing in place of, or
|
||||
/// null — drives the once-per-transition Information log.</summary>
|
||||
private CommonsNodeId? _capturingForUnreachablePrimary;
|
||||
|
||||
/// <summary>The last historization capture verdict published (null before the first), so the reachability
|
||||
/// path republishes only a CHANGED verdict.</summary>
|
||||
private bool? _lastPublishedHistorizationVerdict;
|
||||
|
||||
/// <summary>Whether this actor holds its own (#672) cluster reachability subscription.</summary>
|
||||
private bool _subscribedToReachabilityEvents;
|
||||
|
||||
/// <summary>Gitea #601 — how long an Applied ack waits for the address-space outcome (see
|
||||
/// <see cref="DefaultApplyOutcomeTimeout"/>).</summary>
|
||||
private readonly TimeSpan _applyOutcomeTimeout;
|
||||
@@ -664,7 +688,8 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
|
||||
TimeSpan? reprovisionBackoff = null,
|
||||
TimeSpan? commsRecoveryAuditDelay = null,
|
||||
Func<CommonsNodeId, bool>? sharesAlarmQueueWithLocal = null,
|
||||
TimeSpan? applyOutcomeTimeout = null) =>
|
||||
TimeSpan? applyOutcomeTimeout = null,
|
||||
bool captureWhilePrimaryUnreachable = true) =>
|
||||
// WARNING: this forwarding list is POSITIONAL, and Props.Create compiles it into an
|
||||
// expression tree. Six IActorRef? parameters and several interface-typed ones mean a
|
||||
// mis-ordered argument is usually type-compatible and therefore compiles clean, then binds
|
||||
@@ -677,7 +702,7 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
|
||||
deploymentArtifactCache, redundancyRoleView, replicationPeerHost, fetchAndCacheMode, artifactFetcher,
|
||||
alarmStateStore, telemetryHub, primaryRoleSource, historizationRoleView,
|
||||
sharesHistorizationScopeWithLocal, applyWedgeTimeout, reprovisionBackoff, commsRecoveryAuditDelay,
|
||||
sharesAlarmQueueWithLocal, applyOutcomeTimeout));
|
||||
sharesAlarmQueueWithLocal, applyOutcomeTimeout, captureWhilePrimaryUnreachable));
|
||||
|
||||
/// <summary>Initializes a new DriverHostActor with the specified dependencies.</summary>
|
||||
/// <param name="dbFactory">Database context factory for configuration database access.</param>
|
||||
@@ -745,6 +770,11 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
|
||||
/// PeerAddress), whose partner is its per-cluster mesh partner. Ignored when
|
||||
/// <paramref name="replicationPeerHost"/> is set. Null ⇒ no such peer is known and this node never stands its
|
||||
/// drain down.</param>
|
||||
/// <param name="captureWhilePrimaryUnreachable">Gitea #672 (<c>ContinuousHistorization:CaptureWhilePrimaryUnreachable</c>,
|
||||
/// default <c>true</c>): when a mesh-partner Primary this node would stand its historization capture down for is
|
||||
/// reported UNREACHABLE by the cluster failure detector, stop deferring to it and capture — rather than waiting
|
||||
/// for auto-down + a fresh redundancy snapshot. Affects ONLY the historization capture verdict; the alarm drain
|
||||
/// and the Primary data-plane verdict are unchanged. <c>false</c> restores the pre-#672 behaviour.</param>
|
||||
public DriverHostActor(
|
||||
IDbContextFactory<OtOpcUaConfigDbContext>? dbFactory,
|
||||
CommonsNodeId localNode,
|
||||
@@ -776,8 +806,10 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
|
||||
TimeSpan? reprovisionBackoff = null,
|
||||
TimeSpan? commsRecoveryAuditDelay = null,
|
||||
Func<CommonsNodeId, bool>? sharesAlarmQueueWithLocal = null,
|
||||
TimeSpan? applyOutcomeTimeout = null)
|
||||
TimeSpan? applyOutcomeTimeout = null,
|
||||
bool captureWhilePrimaryUnreachable = true)
|
||||
{
|
||||
_captureWhilePrimaryUnreachable = captureWhilePrimaryUnreachable;
|
||||
_applyOutcomeTimeout = applyOutcomeTimeout ?? DefaultApplyOutcomeTimeout;
|
||||
_sharesAlarmQueueWithLocal = sharesAlarmQueueWithLocal;
|
||||
_applyWedgeTimeout = applyWedgeTimeout ?? DefaultApplyWedgeTimeout;
|
||||
@@ -847,6 +879,7 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
|
||||
// ApplyScriptedAlarms through the same PushDesiredSubscriptions pass.
|
||||
SpawnScriptedAlarmHost();
|
||||
StartPrimaryVerdictPublishing();
|
||||
StartReachabilityTracking();
|
||||
Bootstrap();
|
||||
}
|
||||
|
||||
@@ -894,8 +927,18 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
|
||||
/// <summary>A cluster member changed status: re-publish the Primary verdict now and once more shortly after.
|
||||
/// The deferred pass covers the cluster read-view (what <see cref="DriverMemberCount"/> reads) updating on its
|
||||
/// own subscriber AFTER this event was delivered here — the two are not ordered.</summary>
|
||||
private void OnDriverMembershipChanged(Akka.Cluster.ClusterEvent.IMemberEvent _)
|
||||
private void OnDriverMembershipChanged(Akka.Cluster.ClusterEvent.IMemberEvent evt)
|
||||
{
|
||||
// Gitea #672: a member that comes Up at an address is reachable — the usual case being a restarted
|
||||
// node's new incarnation. A REMOVED member deliberately stays in the set: it never answers again, and
|
||||
// dropping it would re-derive the verdict from a cached snapshot that may still name it Primary,
|
||||
// closing the gate on a dead node until the next snapshot (the very hole #672 closes).
|
||||
if (evt is Akka.Cluster.ClusterEvent.MemberUp up
|
||||
&& _unreachableNodes.Remove(ToNodeId(up.Member.Address)))
|
||||
{
|
||||
RepublishHistorizationVerdictIfChanged();
|
||||
}
|
||||
|
||||
PublishPrimaryVerdictIfChanged();
|
||||
// One keyed timer: a burst of member events collapses into a single deferred pass.
|
||||
Timers.StartSingleTimer("republish-primary-verdict", RepublishPrimaryVerdict.Instance, MembershipRepublishDelay);
|
||||
@@ -919,10 +962,62 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
|
||||
PublishPrimaryVerdict();
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Gitea #672: follow cluster reachability so the historization capture gate stops deferring to a recording
|
||||
/// Primary the failure detector already considers dead. Its OWN subscription, deliberately not folded into
|
||||
/// <see cref="StartPrimaryVerdictPublishing"/>, which early-returns on nodes with no primary-role source.
|
||||
/// <see cref="Akka.Cluster.ClusterEvent.MemberUp"/> rides along so a re-joined incarnation leaves the set.
|
||||
/// </summary>
|
||||
private void StartReachabilityTracking()
|
||||
{
|
||||
if (_historizationRoleView is null || !_captureWhilePrimaryUnreachable) return;
|
||||
try
|
||||
{
|
||||
Akka.Cluster.Cluster.Get(Context.System).Subscribe(
|
||||
Self, Akka.Cluster.ClusterEvent.SubscriptionInitialStateMode.InitialStateAsEvents,
|
||||
typeof(Akka.Cluster.ClusterEvent.ReachabilityEvent),
|
||||
typeof(Akka.Cluster.ClusterEvent.MemberUp));
|
||||
_subscribedToReachabilityEvents = true;
|
||||
}
|
||||
catch (Exception)
|
||||
{
|
||||
// Non-cluster ActorRefProvider — no failure detector to follow; the snapshot path alone decides.
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Gitea #672: a peer became unreachable or reachable again. Records it and re-derives ONLY the
|
||||
/// historization capture verdict from the cached snapshot — the alarm drain and the Primary data-plane verdict
|
||||
/// are deliberately untouched (a false allow there is a dual claimant, not a duplicate sample).</summary>
|
||||
private void OnReachabilityChanged(Akka.Cluster.ClusterEvent.ReachabilityEvent evt)
|
||||
{
|
||||
var node = ToNodeId(evt.Member.Address);
|
||||
if (node == _localNode) return; // this node's own reachability as seen by others is not a peer question
|
||||
var changed = evt is Akka.Cluster.ClusterEvent.UnreachableMember
|
||||
? _unreachableNodes.Add(node)
|
||||
: _unreachableNodes.Remove(node);
|
||||
if (changed) RepublishHistorizationVerdictIfChanged();
|
||||
}
|
||||
|
||||
/// <summary>Re-derives the historization capture verdict from the cached snapshot and publishes it only when it
|
||||
/// differs from the last one published. No-op before the first snapshot.</summary>
|
||||
private void RepublishHistorizationVerdictIfChanged()
|
||||
{
|
||||
if (_historizationRoleView is null || _lastRedundancySnapshotNodes is null) return;
|
||||
var verdict = ComputeHistorizationVerdict(_lastRedundancySnapshotNodes);
|
||||
if (verdict == _lastPublishedHistorizationVerdict) return;
|
||||
_lastPublishedHistorizationVerdict = verdict;
|
||||
_historizationRoleView.PublishHistorizationRole(verdict);
|
||||
}
|
||||
|
||||
/// <summary>Same host:port projection as <c>ClusterRoleInfo</c> / <c>RedundancyStateActor.ToNodeId</c>, so a
|
||||
/// reachability event's member compares equal to the NodeId a redundancy snapshot names.</summary>
|
||||
private static CommonsNodeId ToNodeId(Akka.Actor.Address address) =>
|
||||
CommonsNodeId.Parse($"{address.Host ?? string.Empty}:{address.Port ?? 0}");
|
||||
|
||||
/// <inheritdoc />
|
||||
protected override void PostStop()
|
||||
{
|
||||
if (_subscribedToMemberEvents)
|
||||
if (_subscribedToMemberEvents || _subscribedToReachabilityEvents)
|
||||
{
|
||||
try { Akka.Cluster.Cluster.Get(Context.System).Unsubscribe(Self); }
|
||||
catch (Exception) { /* system shutting down */ }
|
||||
@@ -1349,6 +1444,7 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
|
||||
Receive<RouteNativeAlarmAck>(HandleRouteNativeAlarmAck);
|
||||
Receive<RedundancyStateChanged>(OnRedundancyStateChanged);
|
||||
Receive<Akka.Cluster.ClusterEvent.IMemberEvent>(OnDriverMembershipChanged);
|
||||
Receive<Akka.Cluster.ClusterEvent.ReachabilityEvent>(OnReachabilityChanged);
|
||||
Receive<RepublishPrimaryVerdict>(_ => PublishPrimaryVerdictIfChanged());
|
||||
Receive<Akka.Cluster.ClusterEvent.CurrentClusterState>(_ => PublishPrimaryVerdictIfChanged());
|
||||
// The redundancy-state topic also carries OpcUaProbeResult (OpcUaPublishActor peer-probes).
|
||||
@@ -1410,6 +1506,7 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
|
||||
Receive<RouteNativeAlarmAck>(HandleRouteNativeAlarmAck);
|
||||
Receive<RedundancyStateChanged>(OnRedundancyStateChanged);
|
||||
Receive<Akka.Cluster.ClusterEvent.IMemberEvent>(OnDriverMembershipChanged);
|
||||
Receive<Akka.Cluster.ClusterEvent.ReachabilityEvent>(OnReachabilityChanged);
|
||||
Receive<RepublishPrimaryVerdict>(_ => PublishPrimaryVerdictIfChanged());
|
||||
Receive<Akka.Cluster.ClusterEvent.CurrentClusterState>(_ => PublishPrimaryVerdictIfChanged());
|
||||
// The redundancy-state topic also carries OpcUaProbeResult (OpcUaPublishActor peer-probes).
|
||||
@@ -2208,20 +2305,15 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
|
||||
PrimaryGatePolicy.ShouldDrainAlarmHistory(_localRole, queueSharingPeerIsPrimary: peerIsPrimary));
|
||||
|
||||
// Gitea #537: the continuous-historization CAPTURE verdict. Same role rule as the drain gate
|
||||
// above, but a different peer question — historization is not pair-local, it is fleet-wide by
|
||||
// historian tag name, so the only node worth standing down for is one recording the SAME refs:
|
||||
// a mesh partner serving this node's address space. With no way to establish that (a legacy
|
||||
// node carrying no cluster-scoped role, or a test harness), the predicate is null and this
|
||||
// node keeps capturing. Duplicate samples are recoverable; a hole in the trend is not.
|
||||
var historizationPeerIsPrimary =
|
||||
_sharesHistorizationScopeWithLocal is { } sharesScope
|
||||
&& msg.Nodes.Any(n => n.Role == RedundancyRole.Primary
|
||||
&& n.NodeId != _localNode
|
||||
&& sharesScope(n.NodeId));
|
||||
|
||||
_historizationRoleView?.PublishHistorizationRole(
|
||||
PrimaryGatePolicy.ShouldRecordHistorizedValues(
|
||||
_localRole, addressSpaceSharingPeerIsPrimary: historizationPeerIsPrimary));
|
||||
// above, but a different peer question — see ComputeHistorizationVerdict. Cached (#672) so a
|
||||
// reachability change can re-derive it before the next snapshot arrives.
|
||||
_lastRedundancySnapshotNodes = msg.Nodes;
|
||||
if (_historizationRoleView is not null)
|
||||
{
|
||||
var historizationVerdict = ComputeHistorizationVerdict(msg.Nodes);
|
||||
_lastPublishedHistorizationVerdict = historizationVerdict;
|
||||
_historizationRoleView.PublishHistorizationRole(historizationVerdict);
|
||||
}
|
||||
|
||||
// The DATA-PLANE verdict, deliberately — the same one HandleRouteNodeWrite gates on, not the
|
||||
// drain verdict published immediately above. A driver reading this claims an exclusive identity
|
||||
@@ -2230,6 +2322,66 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
|
||||
PublishPrimaryVerdict();
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The continuous-historization capture verdict for a snapshot (Gitea #537, #672).
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>Historization is not pair-local, it is fleet-wide by historian tag name, so the only node worth
|
||||
/// standing down for is one recording the SAME refs: a mesh partner serving this node's address space. With
|
||||
/// no way to establish that (a legacy node carrying no cluster-scoped role, or a test harness), the
|
||||
/// predicate is null and this node keeps capturing. Duplicate samples are recoverable; a hole in the trend
|
||||
/// is not.</para>
|
||||
/// <para><b>#672 — an UNREACHABLE recording Primary does not count.</b> A crashed (SIGKILLed) Primary stays a
|
||||
/// cluster member until auto-down, and the redundancy singleton usually died with it, so the cached snapshot
|
||||
/// keeps naming it Primary for the whole failure-detector + auto-down window (~25 s) while nobody captures.
|
||||
/// Excluding unreachable peers shrinks that hole to the failure-detector window alone. The cost — a short
|
||||
/// duplicate window, and both nodes capturing for the duration of a genuine partition — is the same
|
||||
/// asymmetric trade #537 already makes. Gated by <c>ContinuousHistorization:CaptureWhilePrimaryUnreachable</c>.</para>
|
||||
/// </remarks>
|
||||
private bool ComputeHistorizationVerdict(IReadOnlyList<NodeRedundancyState> nodes)
|
||||
{
|
||||
var peerIsPrimary = false;
|
||||
CommonsNodeId? unreachablePrimary = null;
|
||||
if (_sharesHistorizationScopeWithLocal is { } sharesScope)
|
||||
{
|
||||
foreach (var n in nodes)
|
||||
{
|
||||
if (n.Role != RedundancyRole.Primary || n.NodeId == _localNode || !sharesScope(n.NodeId)) continue;
|
||||
if (_captureWhilePrimaryUnreachable && _unreachableNodes.Contains(n.NodeId))
|
||||
{
|
||||
unreachablePrimary ??= n.NodeId;
|
||||
continue;
|
||||
}
|
||||
|
||||
peerIsPrimary = true;
|
||||
}
|
||||
}
|
||||
|
||||
var verdict = PrimaryGatePolicy.ShouldRecordHistorizedValues(
|
||||
_localRole, addressSpaceSharingPeerIsPrimary: peerIsPrimary);
|
||||
|
||||
// Log only when the unreachable exclusion is what OPENED the gate — i.e. the verdict would have been
|
||||
// "stand down" had that Primary been reachable.
|
||||
var overriding = unreachablePrimary is not null && verdict
|
||||
&& !PrimaryGatePolicy.ShouldRecordHistorizedValues(_localRole, addressSpaceSharingPeerIsPrimary: true);
|
||||
if (overriding && _capturingForUnreachablePrimary is null)
|
||||
{
|
||||
_capturingForUnreachablePrimary = unreachablePrimary;
|
||||
_log.Info(
|
||||
"DriverHost {Node}: capturing historized values while recording Primary {Primary} is unreachable (Gitea #672)",
|
||||
_localNode, unreachablePrimary);
|
||||
}
|
||||
else if (!overriding && _capturingForUnreachablePrimary is { } previous)
|
||||
{
|
||||
_capturingForUnreachablePrimary = null;
|
||||
_log.Info(
|
||||
"DriverHost {Node}: no longer capturing in place of unreachable recording Primary {Primary}; capture verdict now {Verdict}",
|
||||
_localNode, previous, verdict ? "capture" : "stand down");
|
||||
}
|
||||
|
||||
return verdict;
|
||||
}
|
||||
|
||||
private void Stale()
|
||||
{
|
||||
Receive<DispatchDeployment>(_ =>
|
||||
@@ -2251,6 +2403,7 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
|
||||
Receive<ReconnectDriver>(HandleReconnectDriver);
|
||||
Receive<RedundancyStateChanged>(OnRedundancyStateChanged);
|
||||
Receive<Akka.Cluster.ClusterEvent.IMemberEvent>(OnDriverMembershipChanged);
|
||||
Receive<Akka.Cluster.ClusterEvent.ReachabilityEvent>(OnReachabilityChanged);
|
||||
Receive<RepublishPrimaryVerdict>(_ => PublishPrimaryVerdictIfChanged());
|
||||
Receive<Akka.Cluster.ClusterEvent.CurrentClusterState>(_ => PublishPrimaryVerdictIfChanged());
|
||||
// The redundancy-state topic also carries OpcUaProbeResult (OpcUaPublishActor peer-probes).
|
||||
|
||||
@@ -579,8 +579,8 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
|
||||
/// came back — live-confirmed against a stopped Modbus fixture, which needed a node restart. Both
|
||||
/// comms-loss quality paths (#477 native conditions, #481 scripted-alarm inputs) hang off
|
||||
/// <see cref="ConnectivityChanged"/>, so both were wired to a signal that never fired.
|
||||
/// <c>OpcUaClientDriver</c>'s own reconnect-exhausted branch already assumed this mechanism existed
|
||||
/// ("the Core never learns it must offer an operator reinitialize") — it did not.
|
||||
/// Since Gitea #670 this is also <c>OpcUaClientDriver</c>'s only recovery path for a lost upstream
|
||||
/// session (its in-driver SDK reconnect handler was deleted).
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Why the trip condition is a conjunction, and NOT simply "non-Healthy for N polls".</b>
|
||||
|
||||
@@ -98,6 +98,13 @@ public static class PrimaryGatePolicy
|
||||
/// promoted node takes to see its first snapshot. That is inherent to any non-duplicating
|
||||
/// design; the alternative — both nodes capturing — is what #537 was filed about.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Gitea #672: on a CRASH the newly promoted node may not see a snapshot until auto-down (the
|
||||
/// redundancy singleton died with the old Primary), so the caller excludes a Primary the cluster
|
||||
/// failure detector reports unreachable from <paramref name="addressSpaceSharingPeerIsPrimary"/> —
|
||||
/// shrinking the gap to the failure-detector window. This method is unchanged by that; it only
|
||||
/// sees the narrower input.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
/// <param name="localRole">This node's last-known redundancy role, or <c>null</c> when unknown.</param>
|
||||
/// <param name="addressSpaceSharingPeerIsPrimary">
|
||||
|
||||
@@ -129,6 +129,25 @@ public sealed class ContinuousHistorizationOptions
|
||||
/// <summary>Cap (seconds) on the exponential retry backoff after repeated drain failures. Defaults to 30.</summary>
|
||||
public double MaxBackoffSeconds { get; init; } = 30;
|
||||
|
||||
/// <summary>
|
||||
/// Gitea #672: when <c>true</c> (the default), a redundant node whose capture is stood down for its
|
||||
/// mesh-partner Primary (Gitea #537) resumes capturing as soon as the cluster failure detector reports
|
||||
/// that Primary <b>unreachable</b>, instead of waiting for auto-down and a fresh redundancy snapshot.
|
||||
/// <para>
|
||||
/// A crashed Primary otherwise leaves a capture hole of failure-detection + auto-down (~25 s on the
|
||||
/// shipped <c>akka.conf</c>): the redundancy singleton usually died with it, so the standby's last
|
||||
/// snapshot keeps naming the dead node Primary. With this on, the hole shrinks to the failure-detector
|
||||
/// window (~10–12 s). The cost is duplicates: during a genuine partition both nodes capture from
|
||||
/// detection onward, and a briefly-unreachable-but-alive Primary overlaps the standby. SqlServer and
|
||||
/// QuestDB absorb a duplicate of a source-timestamped sample idempotently; a driver with no source
|
||||
/// timestamp (e.g. Modbus, stamped with each node's own poll time) writes a duplicate row on every
|
||||
/// backend, and the Gateway backend stores duplicates. <c>false</c> restores the pre-#672 behaviour.
|
||||
/// Only the historization capture verdict is affected — never the alarm drain or the Primary data-plane
|
||||
/// gate.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public bool CaptureWhilePrimaryUnreachable { get; init; } = true;
|
||||
|
||||
/// <summary>Returns operator-facing misconfiguration warnings for an <c>Enabled</c> recorder
|
||||
/// (empty when disabled or correctly configured). Pure — the registration logs each entry.</summary>
|
||||
/// <returns>Zero or more human-readable warning messages (never carrying secret values).</returns>
|
||||
|
||||
@@ -0,0 +1,96 @@
|
||||
using System.Diagnostics;
|
||||
using System.Diagnostics.Metrics;
|
||||
using Microsoft.Extensions.Hosting;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Observability;
|
||||
using ZB.MOM.WW.OtOpcUa.Runtime.Redundancy;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Runtime.Historian;
|
||||
|
||||
/// <summary>
|
||||
/// Timer that drives an <see cref="IHistorianRetention"/> purge. Each tick is gated on
|
||||
/// <see cref="IHistorizationRoleView.ShouldRecordHistorizedValues"/> so only the node that is writing
|
||||
/// history purges it (an unknown role allows, matching the capture gate). A null retention (backend has
|
||||
/// none, or no window configured) makes the service inert. Never throws out of the loop.
|
||||
/// </summary>
|
||||
public sealed class HistorianRetentionService : BackgroundService
|
||||
{
|
||||
/// <summary>Counter of rows purged, tagged <c>table</c>.</summary>
|
||||
public const string PurgedCounterName = "otopcua.historian.retention.purged";
|
||||
|
||||
private static readonly Counter<long> Purged = OtOpcUaTelemetry.Meter.CreateCounter<long>(
|
||||
PurgedCounterName, unit: "{row}", description: "Historian rows deleted by the retention purge.");
|
||||
|
||||
private readonly IHistorianRetention? _retention;
|
||||
private readonly IHistorizationRoleView? _roleView;
|
||||
private readonly ILogger _log;
|
||||
private readonly KeyValuePair<string, object?>[] _extraTags;
|
||||
|
||||
/// <summary>Creates the service.</summary>
|
||||
/// <param name="retention">Backend purge seam; null makes the service inert.</param>
|
||||
/// <param name="roleView">Primary gate; null means always run.</param>
|
||||
/// <param name="log">Logger.</param>
|
||||
/// <param name="extraTags">Extra tags added to every counter increment (tests use a unique one).</param>
|
||||
public HistorianRetentionService(IHistorianRetention? retention, IHistorizationRoleView? roleView,
|
||||
ILogger<HistorianRetentionService>? log = null, params KeyValuePair<string, object?>[] extraTags)
|
||||
{
|
||||
_retention = retention;
|
||||
_roleView = roleView;
|
||||
_log = (ILogger?)log ?? NullLogger.Instance;
|
||||
_extraTags = extraTags;
|
||||
}
|
||||
|
||||
/// <summary>Runs one gated tick. Never throws.</summary>
|
||||
/// <param name="ct">Cancellation.</param>
|
||||
/// <returns>True when a purge pass ran.</returns>
|
||||
public async Task<bool> TickAsync(CancellationToken ct)
|
||||
{
|
||||
if (_retention is null) return false;
|
||||
if (_roleView is { ShouldRecordHistorizedValues: false })
|
||||
{
|
||||
_log.LogDebug("Historian retention purge skipped: this node is not the recording node.");
|
||||
return false;
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
var results = await _retention.PurgeAsync(ct).ConfigureAwait(false);
|
||||
foreach (var r in results)
|
||||
{
|
||||
if (r.Rows <= 0) continue;
|
||||
var tags = new TagList { { "table", r.Table } };
|
||||
foreach (var t in _extraTags) tags.Add(t);
|
||||
Purged.Add(r.Rows, tags);
|
||||
_log.LogInformation("Historian retention purged {Rows} rows from {Table}.", r.Rows, r.Table);
|
||||
}
|
||||
return true;
|
||||
}
|
||||
catch (Exception) when (ct.IsCancellationRequested)
|
||||
{
|
||||
// Shutdown: SqlClient surfaces cancellation as SqlException, not only OperationCanceledException.
|
||||
return false;
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_log.LogWarning(ex, "Historian retention purge failed; will retry next interval.");
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
/// <inheritdoc/>
|
||||
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
|
||||
{
|
||||
if (_retention is null) return;
|
||||
try
|
||||
{
|
||||
using var timer = new PeriodicTimer(_retention.Interval);
|
||||
await TickAsync(stoppingToken).ConfigureAwait(false);
|
||||
while (await timer.WaitForNextTickAsync(stoppingToken).ConfigureAwait(false))
|
||||
await TickAsync(stoppingToken).ConfigureAwait(false);
|
||||
}
|
||||
catch (OperationCanceledException) when (stoppingToken.IsCancellationRequested)
|
||||
{
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -39,4 +39,9 @@ public interface IHistorianBackendProvider
|
||||
/// <param name="sp">Service provider.</param>
|
||||
/// <returns>The updater, or <c>null</c> when the backend does not support HistoryUpdate.</returns>
|
||||
IHistorianDataUpdater? CreateUpdater(IServiceProvider sp);
|
||||
|
||||
/// <summary>Creates the retention purge seam, if the backend manages retention in-process.</summary>
|
||||
/// <param name="sp">Service provider.</param>
|
||||
/// <returns>The purger, or <c>null</c> (default) when the backend has no retention or enforces it itself.</returns>
|
||||
IHistorianRetention? CreateRetention(IServiceProvider sp) => null;
|
||||
}
|
||||
|
||||
@@ -0,0 +1,25 @@
|
||||
namespace ZB.MOM.WW.OtOpcUa.Runtime.Historian;
|
||||
|
||||
/// <summary>Rows removed from one historian table by a single purge pass.</summary>
|
||||
/// <param name="Table">Unqualified table name (the <c>table</c> tag on the retention counter).</param>
|
||||
/// <param name="Rows">Rows deleted.</param>
|
||||
public sealed record RetentionPurgeResult(string Table, long Rows);
|
||||
|
||||
/// <summary>
|
||||
/// Backend-side retention purge: deletes history older than the backend's configured retention window.
|
||||
/// Created by <see cref="IHistorianBackendProvider.CreateRetention"/> and driven on a timer by
|
||||
/// <see cref="HistorianRetentionService"/>, which owns scheduling and the Primary gate.
|
||||
/// </summary>
|
||||
public interface IHistorianRetention
|
||||
{
|
||||
/// <summary>How often <see cref="HistorianRetentionService"/> should run a pass.</summary>
|
||||
TimeSpan Interval { get; }
|
||||
|
||||
/// <summary>
|
||||
/// Runs one purge pass. Returns an empty list when the pass was skipped (another process holds the
|
||||
/// purge lock). Implementations honour cancellation between batches.
|
||||
/// </summary>
|
||||
/// <param name="ct">Cancellation.</param>
|
||||
/// <returns>Per-table deleted-row counts.</returns>
|
||||
Task<IReadOnlyList<RetentionPurgeResult>> PurgeAsync(CancellationToken ct);
|
||||
}
|
||||
@@ -112,6 +112,16 @@ public sealed class ServerHistorianOptions
|
||||
/// </summary>
|
||||
public TimeSpan HistoryReadDeadline { get; init; } = TimeSpan.FromSeconds(60);
|
||||
|
||||
/// <summary>
|
||||
/// <c>SqlServer</c> / <c>QuestDb</c> backends only (#671): the most raw in-window samples one Part 13
|
||||
/// time-weighted read (TimeAverage, TimeAverage2, Total, Total2, Interpolative) may stream into the server-side
|
||||
/// evaluator — that family is not pushed down to SQL. A read past the cap stops at cap + 1 rows and fails THAT
|
||||
/// node with <c>BadTooManyOperations</c> instead of holding a pooled connection until
|
||||
/// <see cref="HistoryReadDeadline"/>. Non-positive = unbounded (a <see cref="Validate"/> warning). Default
|
||||
/// 1 000 000 (≈ 11.5 days of a 1 Hz tag).
|
||||
/// </summary>
|
||||
public int MaxTimeWeightedRawRows { get; init; } = 1_000_000;
|
||||
|
||||
/// <summary>Returns operator-facing misconfiguration warnings for an <c>Enabled</c> historian
|
||||
/// (empty when disabled or correctly configured). Pure — the registration logs each entry.
|
||||
/// <para>
|
||||
@@ -154,6 +164,8 @@ public sealed class ServerHistorianOptions
|
||||
warnings.Add($"ServerHistorian:HistoryReadBatchParallelism is {HistoryReadBatchParallelism} — must be > 0; HistoryRead falls back to serving each node sequentially.");
|
||||
if (MaxConcurrentHistoryReadBatches <= 0)
|
||||
warnings.Add($"ServerHistorian:MaxConcurrentHistoryReadBatches is {MaxConcurrentHistoryReadBatches} — must be > 0; the process-wide HistoryRead batch limiter is disabled.");
|
||||
if (MaxTimeWeightedRawRows <= 0)
|
||||
warnings.Add($"ServerHistorian:MaxTimeWeightedRawRows is {MaxTimeWeightedRawRows} — must be > 0; time-weighted aggregate reads on the SqlServer/QuestDb backends stream an unbounded number of raw samples.");
|
||||
if (HistoryReadDeadline <= TimeSpan.Zero)
|
||||
warnings.Add($"ServerHistorian:HistoryReadDeadline is {HistoryReadDeadline} — must be > 0; HistoryRead requests run unbounded (no per-request deadline).");
|
||||
return warnings;
|
||||
|
||||
@@ -625,6 +625,9 @@ public static class ServiceCollectionExtensions
|
||||
// verdict, and how it decides which peer is worth standing down for.
|
||||
historizationRoleView: resolver.GetService<IHistorizationRoleView>(),
|
||||
sharesHistorizationScopeWithLocal: historizationScopePeer,
|
||||
// Gitea #672: stop deferring the capture to a recording Primary the failure detector
|
||||
// already reports unreachable. Unbound section ⇒ the option's default (true).
|
||||
captureWhilePrimaryUnreachable: continuousOptions?.CaptureWhilePrimaryUnreachable ?? true,
|
||||
// Review R6 MINOR-4: the passive half of a LocalDb pair has no PeerAddress, so it is told
|
||||
// its queue-sharing peer as its mesh partner instead — otherwise it never stands down.
|
||||
sharesAlarmQueueWithLocal: BuildAlarmQueuePeerPredicate(
|
||||
|
||||
@@ -326,6 +326,20 @@ public sealed class FakeOpcUaClientService : IOpcUaClientService
|
||||
return Task.FromResult(HistoryReadResult);
|
||||
}
|
||||
|
||||
/// <summary>Gets the list of (NodeId, start, end, maxValues) tuples from HistoryReadModifiedAsync calls.</summary>
|
||||
public List<(NodeId NodeId, DateTime Start, DateTime End, int MaxValues)> HistoryReadModifiedCalls { get; } = [];
|
||||
|
||||
/// <summary>Gets or sets the entries returned by HistoryReadModifiedAsync.</summary>
|
||||
public IReadOnlyList<HistoryModifiedEntry> HistoryReadModifiedResult { get; set; } = [];
|
||||
|
||||
/// <inheritdoc />
|
||||
public Task<IReadOnlyList<HistoryModifiedEntry>> HistoryReadModifiedAsync(
|
||||
NodeId nodeId, DateTime startTime, DateTime endTime, int maxValues = 1000, CancellationToken ct = default)
|
||||
{
|
||||
HistoryReadModifiedCalls.Add((nodeId, startTime, endTime, maxValues));
|
||||
return Task.FromResult(HistoryReadModifiedResult);
|
||||
}
|
||||
|
||||
/// <summary>Gets the details passed to HistoryUpdateAsync.</summary>
|
||||
public List<HistoryUpdateDetails> HistoryUpdateCalls { get; } = [];
|
||||
|
||||
|
||||
@@ -146,4 +146,55 @@ public class HistoryReadCommandTests
|
||||
fakeService.DisconnectCalled.ShouldBeTrue();
|
||||
fakeService.DisposeCalled.ShouldBeTrue();
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>#668: --modified reads the modification trail (not raw) and prints update type, user and prior value.</summary>
|
||||
[Fact]
|
||||
public async Task Execute_ModifiedRead_CallsHistoryReadModified_and_prints_the_trail()
|
||||
{
|
||||
var t = new DateTime(2025, 6, 15, 10, 0, 0, DateTimeKind.Utc);
|
||||
var fakeService = new FakeOpcUaClientService
|
||||
{
|
||||
HistoryReadModifiedResult =
|
||||
[
|
||||
new HistoryModifiedEntry(new DataValue(Variant.Null, StatusCodes.BadNoData, t, t), t.AddDays(1), HistoryUpdateType.Insert, "editor"),
|
||||
new HistoryModifiedEntry(new DataValue(new Variant(1.5), StatusCodes.Good, t, t), t.AddDays(2), HistoryUpdateType.Replace, "editor"),
|
||||
],
|
||||
};
|
||||
var command = new HistoryReadCommand(new FakeOpcUaClientServiceFactory(fakeService))
|
||||
{
|
||||
Url = "opc.tcp://localhost:4840",
|
||||
NodeId = "ns=2;s=HistNode",
|
||||
Modified = true,
|
||||
MaxValues = 50,
|
||||
};
|
||||
|
||||
using var console = TestConsoleHelper.CreateConsole();
|
||||
await command.ExecuteAsync(console);
|
||||
|
||||
fakeService.HistoryReadModifiedCalls.ShouldHaveSingleItem().MaxValues.ShouldBe(50);
|
||||
fakeService.HistoryReadRawCalls.ShouldBeEmpty();
|
||||
var output = TestConsoleHelper.GetOutput(console);
|
||||
output.ShouldContain("Modified history for ns=2;s=HistNode");
|
||||
output.ShouldContain("Replace");
|
||||
output.ShouldContain("Insert");
|
||||
output.ShouldContain("editor");
|
||||
output.ShouldContain("(none)");
|
||||
output.ShouldContain("2 modifications returned.");
|
||||
}
|
||||
|
||||
/// <summary>--modified cannot be combined with --aggregate.</summary>
|
||||
[Fact]
|
||||
public async Task Execute_ModifiedWithAggregate_ThrowsCommandException()
|
||||
{
|
||||
var command = new HistoryReadCommand(new FakeOpcUaClientServiceFactory(new FakeOpcUaClientService()))
|
||||
{
|
||||
Url = "opc.tcp://localhost:4840",
|
||||
NodeId = "ns=2;s=HistNode",
|
||||
Modified = true,
|
||||
Aggregate = "Average",
|
||||
};
|
||||
|
||||
using var console = TestConsoleHelper.CreateConsole();
|
||||
await Should.ThrowAsync<CliFx.Exceptions.CommandException>(async () => await command.ExecuteAsync(console));
|
||||
}
|
||||
}
|
||||
|
||||
@@ -8,6 +8,9 @@ public static class ConformanceData
|
||||
/// <summary>Base time: 2026-10-09 08:00:00.1234567 UTC (sub-millisecond ticks on purpose).</summary>
|
||||
public static readonly DateTime T0 = new DateTime(2026, 10, 9, 8, 0, 0, DateTimeKind.Utc).AddTicks(1_234_567);
|
||||
|
||||
/// <summary>The HistoryUpdate caller the suite passes to every updater call (recorded in the modification trail).</summary>
|
||||
public static readonly ZB.MOM.WW.OtOpcUa.Core.Abstractions.HistoryUpdateContext Editor = new("conformance-editor", "k-conf");
|
||||
|
||||
/// <summary>Good.</summary>
|
||||
public const uint Good = 0x00000000u;
|
||||
/// <summary>Bad (generic).</summary>
|
||||
|
||||
@@ -22,6 +22,10 @@ public interface IConformanceBackend : IAsyncDisposable
|
||||
/// <param name="clock">Its "now"; null means the system clock.</param>
|
||||
IHistorianDataSource CreateDataSource(TimeProvider? clock = null);
|
||||
|
||||
/// <summary>A fresh data source whose time-weighted raw-row cap (<c>ServerHistorian:MaxTimeWeightedRawRows</c>) is <paramref name="maxTimeWeightedRawRows"/>.</summary>
|
||||
/// <param name="maxTimeWeightedRawRows">The cap (<= 0 = unbounded).</param>
|
||||
IHistorianDataSource CreateDataSource(int maxTimeWeightedRawRows);
|
||||
|
||||
/// <summary>A fresh live-value writer over this backend's store.</summary>
|
||||
/// <param name="utcNow">The clock used to stamp values that carry no timestamp; null means the system clock.</param>
|
||||
IHistorianValueWriter CreateValueWriter(Func<DateTime>? utcNow = null);
|
||||
|
||||
@@ -0,0 +1,217 @@
|
||||
using Shouldly;
|
||||
using Xunit;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
|
||||
using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Conformance;
|
||||
|
||||
/// <summary>
|
||||
/// OPC UA Part 11 modified-value history (Gitea #668): every HistoryUpdate a backend applies leaves an ordered
|
||||
/// modification trail — the PRIOR value, the caller's principal, the Part 11 update type and the modification time —
|
||||
/// readable through <see cref="IHistorianDataSource.ReadModifiedAsync"/>. Refused items, live recorder writes and
|
||||
/// reads leave nothing.
|
||||
/// </summary>
|
||||
public abstract class ModifiedValueConformance : ConformanceBase
|
||||
{
|
||||
private static readonly DateTime T0 = new(2026, 1, 1, 0, 0, 0, DateTimeKind.Utc);
|
||||
private static DateTime T(int n) => T0.AddSeconds(n).AddTicks(1230); // µs-aligned so both backends store it exactly
|
||||
private static readonly DateTime Lo = T0.AddHours(-1);
|
||||
private static readonly DateTime Hi = T0.AddHours(1);
|
||||
private static readonly CancellationToken Ct = CancellationToken.None;
|
||||
private static readonly HistoryUpdateContext Editor = ConformanceData.Editor;
|
||||
|
||||
private static HistorizationValue V(int n, double d, uint status = Good) =>
|
||||
new(T(n), HistorianValue.FromDouble(d), status);
|
||||
|
||||
private async Task<IReadOnlyList<HistoryModifiedValue>> TrailAsync(
|
||||
string tag, DateTime? start = null, DateTime? end = null, uint max = 0, HistoryModifiedCursor? after = null)
|
||||
{
|
||||
using var src = B.CreateDataSource();
|
||||
var r = await src.ReadModifiedAsync(tag, start ?? Lo, end ?? Hi, max, after, Ct);
|
||||
if (max == 0) r.HasMore.ShouldBeFalse();
|
||||
return r.Values;
|
||||
}
|
||||
|
||||
private static void ShouldBeNoPrior(HistoryModifiedValue m)
|
||||
{
|
||||
m.Value.Value.ShouldBeNull();
|
||||
m.Value.StatusCode.ShouldBe(HistorianStatusCodes.BadNoData);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Insert_records_one_Insert_per_sample_with_no_prior_value_and_the_principal()
|
||||
{
|
||||
var before = DateTime.UtcNow.AddSeconds(-5);
|
||||
var u = B.CreateUpdater();
|
||||
(await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(2, 2), V(1, 1)], Editor, Ct))
|
||||
.ShouldBe([HistorianStatusCodes.GoodEntryInserted, HistorianStatusCodes.GoodEntryInserted]);
|
||||
|
||||
var trail = await TrailAsync("t");
|
||||
|
||||
trail.Count.ShouldBe(2);
|
||||
trail.Select(m => m.Value.SourceTimestampUtc).ShouldBe([T(1), T(2)]); // ordered by source time
|
||||
trail.ShouldAllBe(m => m.UpdateType == HistoryModificationType.Insert);
|
||||
trail.ShouldAllBe(m => m.UserName == "conformance-editor");
|
||||
trail.ShouldAllBe(m => m.ModificationTimeUtc >= before && m.ModificationTimeUtc <= DateTime.UtcNow.AddSeconds(5));
|
||||
foreach (var m in trail) ShouldBeNoPrior(m);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Replace_records_the_prior_value_and_status_after_the_insert()
|
||||
{
|
||||
var u = B.CreateUpdater();
|
||||
await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(1, 1)], Editor, Ct);
|
||||
var other = new HistoryUpdateContext("second-editor", null);
|
||||
(await u.UpdateDataAsync("t", HistoryUpdateMode.Replace, [V(1, 11, Uncertain)], other, Ct))
|
||||
.ShouldBe([HistorianStatusCodes.GoodEntryReplaced]);
|
||||
(await u.UpdateDataAsync("t", HistoryUpdateMode.Replace, [V(1, 111)], Editor, Ct))
|
||||
.ShouldBe([HistorianStatusCodes.GoodEntryReplaced]);
|
||||
|
||||
var trail = await TrailAsync("t");
|
||||
|
||||
trail.Select(m => m.UpdateType).ShouldBe(
|
||||
[HistoryModificationType.Insert, HistoryModificationType.Replace, HistoryModificationType.Replace]);
|
||||
trail.Select(m => m.UserName).ShouldBe(["conformance-editor", "second-editor", "conformance-editor"]);
|
||||
ShouldBeNoPrior(trail[0]);
|
||||
trail[1].Value.Value.ShouldBe(1d); // what the first Replace overwrote
|
||||
trail[1].Value.StatusCode.ShouldBe(Good);
|
||||
trail[2].Value.Value.ShouldBe(11d); // what the second Replace overwrote, with its status
|
||||
trail[2].Value.StatusCode.ShouldBe(Uncertain);
|
||||
trail.ShouldAllBe(m => m.Value.SourceTimestampUtc == T(1));
|
||||
trail.Select(m => m.ModifiedId).ShouldBeInOrder(SortDirection.Ascending);
|
||||
trail.Select(m => m.ModifiedId).Distinct().Count().ShouldBe(3);
|
||||
// The sample itself carries the latest value.
|
||||
(await B.ReadBackAsync("t")).ShouldHaveSingleItem().V.Number.ShouldBe(111d);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Update_records_Update_with_the_prior_value_when_it_replaced_and_none_when_it_inserted()
|
||||
{
|
||||
var u = B.CreateUpdater();
|
||||
await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(1, 1)], Editor, Ct);
|
||||
(await u.UpdateDataAsync("t", HistoryUpdateMode.Update, [V(2, 2), V(1, 10)], Editor, Ct))
|
||||
.ShouldBe([HistorianStatusCodes.GoodEntryInserted, HistorianStatusCodes.GoodEntryReplaced]);
|
||||
|
||||
var trail = await TrailAsync("t");
|
||||
|
||||
trail.Select(m => (m.Value.SourceTimestampUtc, m.UpdateType)).ShouldBe(
|
||||
[
|
||||
(T(1), HistoryModificationType.Insert),
|
||||
(T(1), HistoryModificationType.Update),
|
||||
(T(2), HistoryModificationType.Update),
|
||||
]);
|
||||
trail[1].Value.Value.ShouldBe(1d);
|
||||
ShouldBeNoPrior(trail[2]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task DeleteAtTime_records_a_Delete_carrying_each_deleted_value()
|
||||
{
|
||||
var u = B.CreateUpdater();
|
||||
await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(1, 1), V(2, 2), V(3, 3, Uncertain)], Editor, Ct);
|
||||
var deleter = new HistoryUpdateContext("deleter", "k-del");
|
||||
(await u.DeleteAtTimeAsync("t", [T(3), T(1), T(9)], deleter, Ct))
|
||||
.ShouldBe([Good, Good, HistorianStatusCodes.BadNoEntryExists]);
|
||||
|
||||
var deletes = (await TrailAsync("t")).Where(m => m.UpdateType == HistoryModificationType.Delete).ToList();
|
||||
|
||||
deletes.Select(m => m.Value.SourceTimestampUtc).ShouldBe([T(1), T(3)]);
|
||||
deletes.Select(m => (double)m.Value.Value!).ShouldBe([1d, 3d]);
|
||||
deletes[1].Value.StatusCode.ShouldBe(Uncertain);
|
||||
deletes.ShouldAllBe(m => m.UserName == "deleter");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task DeleteRaw_records_a_Delete_for_every_sample_it_removed()
|
||||
{
|
||||
var u = B.CreateUpdater();
|
||||
await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(1, 1), V(2, 2), V(3, 3)], Editor, Ct);
|
||||
(await u.DeleteRawAsync("t", T(1), T(3), Editor, Ct)).ShouldBe(Good); // [T1, T3)
|
||||
|
||||
var deletes = (await TrailAsync("t")).Where(m => m.UpdateType == HistoryModificationType.Delete).ToList();
|
||||
|
||||
deletes.Select(m => m.Value.SourceTimestampUtc).ShouldBe([T(1), T(2)]);
|
||||
deletes.Select(m => (double)m.Value.Value!).ShouldBe([1d, 2d]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Refused_items_leave_no_trail()
|
||||
{
|
||||
var u = B.CreateUpdater();
|
||||
await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(1, 1)], Editor, Ct);
|
||||
(await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(1, 99)], Editor, Ct)).ShouldBe([HistorianStatusCodes.BadEntryExists]);
|
||||
(await u.UpdateDataAsync("t", HistoryUpdateMode.Replace, [V(5, 5)], Editor, Ct)).ShouldBe([HistorianStatusCodes.BadNoEntryExists]);
|
||||
(await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [new HistorizationValue(null, HistorianValue.FromDouble(1), Good)], Editor, Ct))
|
||||
.ShouldBe([HistorianStatusCodes.BadInvalidTimestamp]);
|
||||
(await u.DeleteAtTimeAsync("t", [T(7)], Editor, Ct)).ShouldBe([HistorianStatusCodes.BadNoEntryExists]);
|
||||
(await u.DeleteRawAsync("t", T(10), T(20), Editor, Ct)).ShouldBe(HistorianStatusCodes.BadNoData);
|
||||
|
||||
var trail = await TrailAsync("t");
|
||||
trail.ShouldHaveSingleItem().UpdateType.ShouldBe(HistoryModificationType.Insert); // only the first, applied insert
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Live_recorder_writes_leave_no_trail()
|
||||
{
|
||||
var w = B.CreateValueWriter();
|
||||
(await w.WriteLiveValuesAsync("live", [new(T(1), HistorianValue.FromDouble(1), Good), new(T(2), HistorianValue.FromDouble(2), Good)], Ct))
|
||||
.ShouldBeTrue();
|
||||
(await B.ReadBackAsync("live")).Count.ShouldBe(2); // the writes landed …
|
||||
(await TrailAsync("live")).ShouldBeEmpty(); // … and recorded no modification
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task An_unknown_tag_has_an_empty_trail_and_reads_create_no_tag()
|
||||
{
|
||||
(await TrailAsync("never")).ShouldBeEmpty();
|
||||
(await B.TagExistsAsync("never")).ShouldBeFalse();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task The_trail_pages_by_key_exactly_once_and_reports_HasMore()
|
||||
{
|
||||
var u = B.CreateUpdater();
|
||||
await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(1, 1), V(2, 2), V(3, 3)], Editor, Ct);
|
||||
await u.UpdateDataAsync("t", HistoryUpdateMode.Replace, [V(2, 20)], Editor, Ct);
|
||||
await u.UpdateDataAsync("t", HistoryUpdateMode.Replace, [V(2, 200)], Editor, Ct);
|
||||
var all = await TrailAsync("t");
|
||||
all.Count.ShouldBe(5); // T1 ins | T2 ins, rep, rep | T3 ins
|
||||
|
||||
var seen = new List<long>();
|
||||
HistoryModifiedCursor? cursor = null;
|
||||
using var src = B.CreateDataSource();
|
||||
for (var page = 0; page < 10; page++)
|
||||
{
|
||||
var r = await src.ReadModifiedAsync("t", Lo, Hi, 2, cursor, Ct);
|
||||
seen.AddRange(r.Values.Select(m => m.ModifiedId));
|
||||
if (!r.HasMore) break;
|
||||
r.Values.Count.ShouldBe(2);
|
||||
var last = r.Values[^1];
|
||||
cursor = new HistoryModifiedCursor(last.Value.SourceTimestampUtc!.Value, last.ModifiedId);
|
||||
}
|
||||
|
||||
seen.ShouldBe(all.Select(m => m.ModifiedId)); // a page boundary inside the T2 run neither repeats nor skips
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task A_reverse_window_reads_the_trail_descending()
|
||||
{
|
||||
var u = B.CreateUpdater();
|
||||
await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(1, 1), V(2, 2)], Editor, Ct);
|
||||
await u.UpdateDataAsync("t", HistoryUpdateMode.Replace, [V(2, 20)], Editor, Ct);
|
||||
var forward = await TrailAsync("t");
|
||||
|
||||
var reverse = await TrailAsync("t", start: Hi, end: Lo);
|
||||
|
||||
reverse.Select(m => m.ModifiedId).ShouldBe(forward.Select(m => m.ModifiedId).Reverse());
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task The_window_is_half_open_on_source_time()
|
||||
{
|
||||
var u = B.CreateUpdater();
|
||||
await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(1, 1), V(2, 2), V(3, 3)], Editor, Ct);
|
||||
(await TrailAsync("t", start: T(1), end: T(3))).Select(m => m.Value.SourceTimestampUtc).ShouldBe([T(1), T(2)]);
|
||||
(await TrailAsync("t", start: T(2), end: T(2))).Select(m => m.Value.SourceTimestampUtc).ShouldBe([T(2)]);
|
||||
}
|
||||
}
|
||||
@@ -39,7 +39,6 @@ public abstract class ProcessedConformance : ConformanceBase
|
||||
[InlineData(HistoryAggregateType.Average, 15d, 40d, 50d)]
|
||||
[InlineData(HistoryAggregateType.Minimum, 10d, 40d, 50d)]
|
||||
[InlineData(HistoryAggregateType.Maximum, 20d, 40d, 50d)]
|
||||
[InlineData(HistoryAggregateType.Total, 30d, 40d, 50d)]
|
||||
[InlineData(HistoryAggregateType.Start, 10d, 40d, 50d)]
|
||||
[InlineData(HistoryAggregateType.End, 20d, 40d, 50d)]
|
||||
[InlineData(HistoryAggregateType.Delta, 10d, 0d, 0d)]
|
||||
@@ -121,8 +120,8 @@ public abstract class ProcessedConformance : ConformanceBase
|
||||
(edge, HistorianValue.FromDouble(4), Good), // first tick of B1
|
||||
(At(40).AddTicks(-B.ResolutionTicks), HistorianValue.FromDouble(8), Good), // last tick of B1
|
||||
(At(40), HistorianValue.FromDouble(2000), Good)); // == end: excluded
|
||||
var s = (await B.CreateDataSource().ReadProcessedAsync("E", At(0), At(40), Twenty, HistoryAggregateType.Total, Ct)).Samples;
|
||||
s.Select(x => x.Value).ShouldBe([3d, 12d]);
|
||||
var s = (await B.CreateDataSource().ReadProcessedAsync("E", At(0), At(40), Twenty, HistoryAggregateType.Average, Ct)).Samples;
|
||||
s.Select(x => x.Value).ShouldBe([1.5d, 6d]);
|
||||
s.ShouldAllBe(x => x.StatusCode == GoodCalc);
|
||||
}
|
||||
|
||||
@@ -164,11 +163,15 @@ public abstract class ProcessedConformance : ConformanceBase
|
||||
(await src.ReadProcessedAsync("A", start, end, TimeSpan.Zero, HistoryAggregateType.Count, Ct))
|
||||
.Samples.Single().Value.ShouldBe(4);
|
||||
// Also a whole-DateTime-range window (0001 → 9999-12-31T23:59:59.9999999).
|
||||
(await src.ReadProcessedAsync("A", DateTime.MinValue, DateTime.MaxValue, TimeSpan.Zero, HistoryAggregateType.Total, Ct))
|
||||
.Samples.Single().Value.ShouldBe(120d);
|
||||
(await src.ReadProcessedAsync("A", DateTime.MinValue, DateTime.MaxValue, TimeSpan.Zero, HistoryAggregateType.Count, Ct))
|
||||
.Samples.Single().Value.ShouldBe(4);
|
||||
// An interval wider than the window (here TimeSpan.MaxValue) is the same single interval, no overflow.
|
||||
(await src.ReadProcessedAsync("A", start, end, TimeSpan.MaxValue, HistoryAggregateType.Total, Ct))
|
||||
.Samples.Single().Value.ShouldBe(120d);
|
||||
(await src.ReadProcessedAsync("A", start, end, TimeSpan.MaxValue, HistoryAggregateType.Count, Ct))
|
||||
.Samples.Single().Value.ShouldBe(4);
|
||||
// The time-weighted path over the whole range: ≈ the extrapolated 50 that dominates eight millennia.
|
||||
var tw = (await src.ReadProcessedAsync("A", start, end, TimeSpan.Zero, HistoryAggregateType.TimeAverage2, Ct)).Samples.Single();
|
||||
((double)tw.Value!).ShouldBe(50d, 1e-6);
|
||||
tw.StatusCode.ShouldBe(UncertainCalc);
|
||||
src.GetHealthSnapshot().TotalFailures.ShouldBe(0);
|
||||
}
|
||||
|
||||
@@ -303,7 +306,8 @@ public abstract class ProcessedConformance : ConformanceBase
|
||||
async Task<DataValueSnapshot> One(HistoryAggregateType a) =>
|
||||
(await src.ReadProcessedAsync("B", At(0), At(20), Twenty, a, Ct)).Samples.Single();
|
||||
(await One(HistoryAggregateType.Average)).Value.ShouldBe(0.75d);
|
||||
(await One(HistoryAggregateType.Total)).Value.ShouldBe(3d);
|
||||
// Total is time-weighted (#671): stepped 1 over [0,5), 0 over [5,10), 1 over [10,20) ⇒ 15 value·s.
|
||||
(await One(HistoryAggregateType.Total)).Value.ShouldBe(15d);
|
||||
(await One(HistoryAggregateType.Start)).Value.ShouldBe(true);
|
||||
(await One(HistoryAggregateType.Count)).Value.ShouldBe(4);
|
||||
}
|
||||
@@ -317,8 +321,6 @@ public abstract class ProcessedConformance : ConformanceBase
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData(HistoryAggregateType.Interpolative)]
|
||||
[InlineData(HistoryAggregateType.TimeAverage)]
|
||||
[InlineData(HistoryAggregateType.StartBound)]
|
||||
[InlineData(HistoryAggregateType.EndBound)]
|
||||
[InlineData(HistoryAggregateType.DeltaBounds)]
|
||||
@@ -338,6 +340,174 @@ public abstract class ProcessedConformance : ConformanceBase
|
||||
src.GetHealthSnapshot().TotalQueries.ShouldBe(0);
|
||||
}
|
||||
|
||||
// ------------------------------------------------------------------------------------------ Time-weighted (#671)
|
||||
|
||||
private const uint GoodRaw = 0x00000000u;
|
||||
private const uint GoodInterp = 0x00000402u; // Good | InfoType 0x400 | Interpolated 0x2
|
||||
private const uint UncertainInterp = 0x40A40402u; // UncertainDataSubNormal | 0x402
|
||||
private const uint NoData = 0x809B0000u;
|
||||
private const uint BadCalc = 0x80000401u; // generic Bad | Calculated (an all-outage interval)
|
||||
|
||||
/// <summary>
|
||||
/// Sparse tag W: −30 = 0 G (bound before the window), +20 = 40 G, +30 = outage marker, +50 = 20 G, +100 = 20 G
|
||||
/// (bound after the window). Window [t0, t0+80s). Simple-bounding signal: [−30,20) 0→40 sloped Good (24 at +0,
|
||||
/// 32 at +10); [20,30) 40 held into the outage, Uncertain; [30,50) Bad; [50,100) 20 Good. Interpolated bounding:
|
||||
/// the same, but [20,50) is sloped 40→20 straight across the outage (100/3 at +30, 80/3 at +40).
|
||||
/// </summary>
|
||||
private Task SeedWAsync() => B.SeedAsync("W",
|
||||
(At(-30), HistorianValue.FromDouble(0), Good),
|
||||
(At(20), HistorianValue.FromDouble(40), Good),
|
||||
(At(30), HistorianValue.Empty, BadNoCommunication),
|
||||
(At(50), HistorianValue.FromDouble(20), Good),
|
||||
(At(100), HistorianValue.FromDouble(20), Good));
|
||||
|
||||
private async Task<IReadOnlyList<DataValueSnapshot>> ReadW(HistoryAggregateType agg, double intervalS) =>
|
||||
(await B.CreateDataSource().ReadProcessedAsync("W", At(0), At(80), TimeSpan.FromSeconds(intervalS), agg, Ct)).Samples;
|
||||
|
||||
private static void Values(IReadOnlyList<DataValueSnapshot> s, params double?[] expected)
|
||||
{
|
||||
s.Count.ShouldBe(expected.Length);
|
||||
for (var i = 0; i < expected.Length; i++)
|
||||
{
|
||||
if (expected[i] is { } e) ((double)s[i].Value!).ShouldBe(e, 1e-9, $"bucket {i}");
|
||||
else s[i].Value.ShouldBeNull($"bucket {i}");
|
||||
}
|
||||
}
|
||||
|
||||
[Theory]
|
||||
// 40 s intervals (wider than the sample gaps). B0 [0,40): raw [0,20) 24→40 = 640 Good + [20,30) 40 = 400 Uncertain,
|
||||
// [30,40) Bad; bridged 640 + [20,40) 40→80/3 = 2000/3. B1 [40,80): raw [50,80) 20 = 600 Good, [40,50) Bad; bridged
|
||||
// [40,50) 80/3→20 = 700/3 + 600.
|
||||
[InlineData(HistoryAggregateType.TimeAverage, 98d / 3d, 125d / 6d)] // (3920/3)/40, (2500/3)/40
|
||||
[InlineData(HistoryAggregateType.Total, 3920d / 3d, 2500d / 3d)] // TimeAverage × 40 s
|
||||
[InlineData(HistoryAggregateType.TimeAverage2, 104d / 3d, 20d)] // 1040/30 s, 600/30 s
|
||||
[InlineData(HistoryAggregateType.Total2, 1040d, 600d)]
|
||||
public async Task Time_weighted_aggregates_over_a_sparse_tag_with_wide_intervals(HistoryAggregateType agg, double b0, double b1)
|
||||
{
|
||||
await SeedWAsync();
|
||||
var s = await ReadW(agg, 40);
|
||||
Values(s, b0, b1);
|
||||
// Neither interval is Good throughout (an outage in each) ⇒ Uncertain.
|
||||
s.Select(x => x.StatusCode).ShouldBe([UncertainCalc, UncertainCalc]);
|
||||
s.Select(x => x.SourceTimestampUtc).ShouldBe([B.Round(At(0)), B.Round(At(40))]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Interpolative_over_a_sparse_tag_with_wide_intervals()
|
||||
{
|
||||
await SeedWAsync();
|
||||
var s = await ReadW(HistoryAggregateType.Interpolative, 40);
|
||||
// +0: from the bound before the window, 0 + 40·(30/50) = 24, Good. +40: inside the outage, bridged 80/3, Uncertain.
|
||||
Values(s, 24, 80d / 3d);
|
||||
s.Select(x => x.StatusCode).ShouldBe([GoodInterp, UncertainInterp]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task TimeAverage2_over_a_sparse_tag_with_narrow_intervals()
|
||||
{
|
||||
await SeedWAsync();
|
||||
// 10 s intervals (narrower than the gaps): [0,10) 24→32 = 28; [10,20) 32→40 = 36; [20,30) 40 held up to the
|
||||
// outage (Uncertain time, which counts as Good: TreatUncertainAsBad = false); [30,40) and [40,50) inside the outage
|
||||
// ⇒ generic Bad (data, but none non-Bad); [50,60) … [70,80) the flat 20 up to the bound after, Good.
|
||||
var s = await ReadW(HistoryAggregateType.TimeAverage2, 10);
|
||||
Values(s, 28, 36, 40, null, null, 20, 20, 20);
|
||||
s.Select(x => x.StatusCode).ShouldBe(
|
||||
[GoodCalc, GoodCalc, GoodCalc, BadCalc, BadCalc, GoodCalc, GoodCalc, GoodCalc]);
|
||||
// Total2 of the first interval: 28 × 10 s.
|
||||
((double)(await ReadW(HistoryAggregateType.Total2, 10))[0].Value!).ShouldBe(280d, 1e-9);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Interpolative_over_a_sparse_tag_with_narrow_intervals()
|
||||
{
|
||||
await SeedWAsync();
|
||||
var s = await ReadW(HistoryAggregateType.Interpolative, 10);
|
||||
// +0 24, +10 32 (sloped from the bound); +20 exact 40 (raw Good); +30 ON the outage marker ⇒ bridged 100/3;
|
||||
// +40 bridged 80/3; +50 exact 20; +60, +70 on the Good 20→20 segment to the bound after.
|
||||
Values(s, 24, 32, 40, 100d / 3d, 80d / 3d, 20, 20, 20);
|
||||
s.Select(x => x.StatusCode).ShouldBe(
|
||||
[GoodInterp, GoodInterp, GoodRaw, UncertainInterp, UncertainInterp, GoodRaw, GoodInterp, GoodInterp]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task A_reverse_Interpolative_read_is_anchored_at_its_later_time()
|
||||
{
|
||||
await SeedWAsync();
|
||||
// At(80) → At(0), 40 s: points At(80) (on the Good 20→20 segment to the bound at +100 ⇒ 20) and At(40) (inside
|
||||
// the outage ⇒ bridged 80/3, Uncertain), latest first. A forward read would answer At(0) and At(40) instead.
|
||||
var s = (await B.CreateDataSource().ReadProcessedAsync("W", At(80), At(0), TimeSpan.FromSeconds(40),
|
||||
HistoryAggregateType.Interpolative, Ct)).Samples;
|
||||
s.Select(x => x.SourceTimestampUtc).ShouldBe([B.Round(At(80)), B.Round(At(40))]);
|
||||
Values(s, 20, 80d / 3d);
|
||||
s.Select(x => x.StatusCode).ShouldBe([GoodInterp, UncertainInterp]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task The_time_weighted_raw_row_cap_fails_the_read_past_the_cap_only()
|
||||
{
|
||||
// Tag A holds five samples in [0,80); W holds three in [0,80) plus two bound rows outside it.
|
||||
await ConformanceData.SeedAAsync(B);
|
||||
await SeedWAsync();
|
||||
var atCap = B.CreateDataSource(5);
|
||||
(await atCap.ReadProcessedAsync("A", At(0), At(80), Twenty, HistoryAggregateType.Total, Ct)).Samples.Count.ShouldBe(4);
|
||||
|
||||
var under = B.CreateDataSource(4);
|
||||
var ex = await Should.ThrowAsync<HistoryReadLimitExceededException>(
|
||||
() => under.ReadProcessedAsync("A", At(0), At(80), Twenty, HistoryAggregateType.Total, Ct));
|
||||
ex.Limit.ShouldBe(4);
|
||||
// Pushed-down (per-sample) aggregates are not capped; the bound rows outside the window do not count.
|
||||
(await under.ReadProcessedAsync("A", At(0), At(80), Twenty, HistoryAggregateType.Average, Ct)).Samples.Count.ShouldBe(4);
|
||||
(await B.CreateDataSource(3).ReadProcessedAsync("W", At(0), At(80), Twenty, HistoryAggregateType.TimeAverage, Ct))
|
||||
.Samples.Count.ShouldBe(4);
|
||||
// A client-request limit, not a backend fault.
|
||||
under.GetHealthSnapshot().TotalFailures.ShouldBe(0);
|
||||
// Non-positive = unbounded.
|
||||
(await B.CreateDataSource(0).ReadProcessedAsync("A", At(0), At(80), Twenty, HistoryAggregateType.Total, Ct))
|
||||
.Samples.Count.ShouldBe(4);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Outages_straddling_both_window_edges_use_the_four_bound_seeks()
|
||||
{
|
||||
// −10 = 0 G, −5 outage, +10 = 20 G, +20 outage, +30 = 40 G; window [0,20). The four seeks return −10 (latest
|
||||
// usable before), −5 (latest before), +20 (first at/after end) and +30 (first usable at/after end).
|
||||
await B.SeedAsync("O",
|
||||
(At(-10), HistorianValue.FromDouble(0), Good),
|
||||
(At(-5), HistorianValue.Empty, BadNoCommunication),
|
||||
(At(10), HistorianValue.FromDouble(20), Good),
|
||||
(At(20), HistorianValue.Empty, BadNoCommunication),
|
||||
(At(30), HistorianValue.FromDouble(40), Good));
|
||||
var src = B.CreateDataSource();
|
||||
async Task<IReadOnlyList<DataValueSnapshot>> Read(HistoryAggregateType a, double intervalS) =>
|
||||
(await src.ReadProcessedAsync("O", At(0), At(20), TimeSpan.FromSeconds(intervalS), a, Ct)).Samples;
|
||||
|
||||
// Bridged: −10..10 0→20 ⇒ [0,10) mean 15 (150); 10..30 20→40 ⇒ [10,20) mean 25 (250). 400 / 20 = 20.
|
||||
// (Without the usable-before seek: 250/10 = 25; without the usable-after seek: [10,20) held 20 ⇒ 17.5.)
|
||||
var ta = (await Read(HistoryAggregateType.TimeAverage, 0)).Single();
|
||||
((double)ta.Value!).ShouldBe(20d, 1e-9);
|
||||
ta.StatusCode.ShouldBe(UncertainCalc);
|
||||
// Simple bounding: [0,10) inside the outage from −5; [10,20) 20 held into the outage at +20 ⇒ 200 / 10 s = 20.
|
||||
// (Without the any-before seek: (150+200)/20 = 17.5; without the any-after seek: [10,20) 20→30 ⇒ 25.)
|
||||
var ta2 = (await Read(HistoryAggregateType.TimeAverage2, 0)).Single();
|
||||
((double)ta2.Value!).ShouldBe(20d, 1e-9);
|
||||
ta2.StatusCode.ShouldBe(UncertainCalc);
|
||||
// +0 inside the outage ⇒ bridged 10, Uncertain; +10 exact 20, raw Good.
|
||||
var i = await Read(HistoryAggregateType.Interpolative, 10);
|
||||
Values(i, 10, 20);
|
||||
i.Select(x => x.StatusCode).ShouldBe([UncertainInterp, GoodRaw]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Time_weighted_aggregates_on_an_unknown_tag_are_BadNoData()
|
||||
{
|
||||
var src = B.CreateDataSource();
|
||||
(await src.ReadProcessedAsync("nope", At(0), At(40), Twenty, HistoryAggregateType.TimeAverage, Ct))
|
||||
.Samples.ShouldAllBe(x => x.StatusCode == NoDataCalc);
|
||||
(await src.ReadProcessedAsync("nope", At(0), At(40), Twenty, HistoryAggregateType.Interpolative, Ct))
|
||||
.Samples.ShouldAllBe(x => x.StatusCode == NoData);
|
||||
src.GetHealthSnapshot().TotalFailures.ShouldBe(0);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task The_bucket_count_is_capped_at_100000()
|
||||
{
|
||||
|
||||
@@ -85,7 +85,7 @@ public abstract class QualityContractConformance : ConformanceBase
|
||||
new(At(1), HistorianValue.Empty, 0x40000000u),
|
||||
new(At(2), HistorianValue.Empty, BadNoCommunication),
|
||||
new(At(3), HistorianValue.FromDouble(1), 0u),
|
||||
], Ct);
|
||||
], ConformanceData.Editor, Ct);
|
||||
r.ShouldBe([BadInvalidArgument, BadInvalidArgument, HistorianStatusCodes.GoodEntryInserted,
|
||||
HistorianStatusCodes.GoodEntryInserted]);
|
||||
(await B.CreateDataSource().ReadRawAsync("u", At(0), At(10), 0, Ct)).Samples
|
||||
|
||||
@@ -40,6 +40,22 @@ public abstract class RawReadConformance : ConformanceBase
|
||||
Values(await B.CreateDataSource().ReadRawAsync("A", At(-1), At(100), uint.MaxValue, Ct)).Length.ShouldBe(5);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// #669: the node manager now pages an open-ended read (Part 11: only a StartTime / only an EndTime) with
|
||||
/// <c>DateTime.MaxValue</c> / <c>DateTime.MinValue</c> as the missing bound, so every backend must bind both
|
||||
/// extremes without overflow, ±infinity or a cast error.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public async Task Extreme_bounds_MaxValue_and_MinValue_are_safe()
|
||||
{
|
||||
await ConformanceData.SeedAAsync(B);
|
||||
var max = new DateTime(DateTime.MaxValue.Ticks, DateTimeKind.Utc);
|
||||
var min = new DateTime(0, DateTimeKind.Utc);
|
||||
Values(await B.CreateDataSource().ReadRawAsync("A", At(10), max, 2, Ct)).ShouldBe([20d, 30d]);
|
||||
Values(await B.CreateDataSource().ReadRawAsync("A", At(100), min, 2, Ct)).ShouldBe([50d, 40d]);
|
||||
Values(await B.CreateDataSource().ReadRawAsync("A", At(10), max, 0, Ct)).ShouldBe([20d, 30d, 40d, 50d]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Reverse_read_is_descending_over_end_exclusive_start_inclusive()
|
||||
{
|
||||
|
||||
@@ -39,10 +39,10 @@ public abstract class TagNameConformance : ConformanceBase
|
||||
var u = B.CreateUpdater();
|
||||
|
||||
(await u.UpdateDataAsync(Long, HistoryUpdateMode.Update,
|
||||
[new(At(0), HistorianValue.FromDouble(9), 0u), new(At(1), HistorianValue.FromDouble(9), 0u)], Ct))
|
||||
[new(At(0), HistorianValue.FromDouble(9), 0u), new(At(1), HistorianValue.FromDouble(9), 0u)], ConformanceData.Editor, Ct))
|
||||
.ShouldBe([BadInvalidArgument, BadInvalidArgument]);
|
||||
(await u.DeleteRawAsync(Long, At(-1), At(10), Ct)).ShouldBe(BadInvalidArgument);
|
||||
(await u.DeleteAtTimeAsync(Long, [At(0)], Ct)).ShouldBe([BadInvalidArgument]);
|
||||
(await u.DeleteRawAsync(Long, At(-1), At(10), ConformanceData.Editor, Ct)).ShouldBe(BadInvalidArgument);
|
||||
(await u.DeleteAtTimeAsync(Long, [At(0)], ConformanceData.Editor, Ct)).ShouldBe([BadInvalidArgument]);
|
||||
|
||||
// The prefix series is untouched: not replaced, not deleted.
|
||||
var raw = (await B.CreateDataSource().ReadRawAsync(Prefix, At(-1), At(10), 0, Ct)).Samples;
|
||||
|
||||
@@ -23,10 +23,10 @@ public abstract class UpdateConformance : ConformanceBase
|
||||
public async Task Insert_new_is_GoodEntryInserted_and_existing_is_BadEntryExists_in_order()
|
||||
{
|
||||
var u = B.CreateUpdater();
|
||||
var r1 = await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T(1), 1), V(T(2), 2)], Ct);
|
||||
var r1 = await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T(1), 1), V(T(2), 2)], ConformanceData.Editor, Ct);
|
||||
r1.ShouldBe([HistorianStatusCodes.GoodEntryInserted, HistorianStatusCodes.GoodEntryInserted]);
|
||||
// Mixed: T(2) exists, T(3) new, T(1) exists - out of timestamp order on purpose.
|
||||
var r2 = await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T(3), 3), V(T(2), 99), V(T(1), 98)], Ct);
|
||||
var r2 = await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T(3), 3), V(T(2), 99), V(T(1), 98)], ConformanceData.Editor, Ct);
|
||||
r2.ShouldBe([HistorianStatusCodes.GoodEntryInserted, HistorianStatusCodes.BadEntryExists, HistorianStatusCodes.BadEntryExists]);
|
||||
var rows = await B.ReadBackAsync("t");
|
||||
rows.Select(x => (double?)x.V.Number).ShouldBe([1d, 2d, 3d]); // existing values untouched
|
||||
@@ -37,9 +37,9 @@ public abstract class UpdateConformance : ConformanceBase
|
||||
public async Task Replace_existing_is_GoodEntryReplaced_with_new_value_and_status_missing_is_BadNoEntryExists()
|
||||
{
|
||||
var u = B.CreateUpdater();
|
||||
await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T(1), 1), V(T(2), 2)], Ct);
|
||||
await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T(1), 1), V(T(2), 2)], ConformanceData.Editor, Ct);
|
||||
var r = await u.UpdateDataAsync("t", HistoryUpdateMode.Replace,
|
||||
[V(T(2), 22, UncertainStatus), V(T(5), 5), V(T(1), 11)], Ct);
|
||||
[V(T(2), 22, UncertainStatus), V(T(5), 5), V(T(1), 11)], ConformanceData.Editor, Ct);
|
||||
r.ShouldBe([HistorianStatusCodes.GoodEntryReplaced, HistorianStatusCodes.BadNoEntryExists, HistorianStatusCodes.GoodEntryReplaced]);
|
||||
var rows = await B.ReadBackAsync("t");
|
||||
rows.Count.ShouldBe(2); // T(5) was not created
|
||||
@@ -53,8 +53,8 @@ public abstract class UpdateConformance : ConformanceBase
|
||||
public async Task Update_upserts_inserted_or_replaced_per_item()
|
||||
{
|
||||
var u = B.CreateUpdater();
|
||||
await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T(1), 1)], Ct);
|
||||
var r = await u.UpdateDataAsync("t", HistoryUpdateMode.Update, [V(T(2), 2), V(T(1), 10)], Ct);
|
||||
await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T(1), 1)], ConformanceData.Editor, Ct);
|
||||
var r = await u.UpdateDataAsync("t", HistoryUpdateMode.Update, [V(T(2), 2), V(T(1), 10)], ConformanceData.Editor, Ct);
|
||||
r.ShouldBe([HistorianStatusCodes.GoodEntryInserted, HistorianStatusCodes.GoodEntryReplaced]);
|
||||
(await B.ReadBackAsync("t")).Select(x => (double?)x.V.Number).ShouldBe([10d, 2d]);
|
||||
}
|
||||
@@ -63,15 +63,15 @@ public abstract class UpdateConformance : ConformanceBase
|
||||
public async Task Unknown_tag_Insert_and_Update_create_it_but_Replace_and_deletes_do_not()
|
||||
{
|
||||
var u = B.CreateUpdater();
|
||||
(await u.UpdateDataAsync("rep", HistoryUpdateMode.Replace, [V(T(1), 1)], Ct))
|
||||
(await u.UpdateDataAsync("rep", HistoryUpdateMode.Replace, [V(T(1), 1)], ConformanceData.Editor, Ct))
|
||||
.ShouldBe([HistorianStatusCodes.BadNoEntryExists]);
|
||||
(await u.DeleteRawAsync("del", T(0), T(9), Ct)).ShouldBe(HistorianStatusCodes.BadNoData);
|
||||
(await u.DeleteAtTimeAsync("del2", [T(1)], Ct)).ShouldBe([HistorianStatusCodes.BadNoEntryExists]);
|
||||
(await u.DeleteRawAsync("del", T(0), T(9), ConformanceData.Editor, Ct)).ShouldBe(HistorianStatusCodes.BadNoData);
|
||||
(await u.DeleteAtTimeAsync("del2", [T(1)], ConformanceData.Editor, Ct)).ShouldBe([HistorianStatusCodes.BadNoEntryExists]);
|
||||
foreach (var name in new[] { "rep", "del", "del2" })
|
||||
(await B.TagExistsAsync(name)).ShouldBeFalse(name); // none of those created a tag
|
||||
|
||||
(await u.UpdateDataAsync("ins", HistoryUpdateMode.Insert, [V(T(1), 1)], Ct)).ShouldBe([HistorianStatusCodes.GoodEntryInserted]);
|
||||
(await u.UpdateDataAsync("upd", HistoryUpdateMode.Update, [V(T(1), 1)], Ct)).ShouldBe([HistorianStatusCodes.GoodEntryInserted]);
|
||||
(await u.UpdateDataAsync("ins", HistoryUpdateMode.Insert, [V(T(1), 1)], ConformanceData.Editor, Ct)).ShouldBe([HistorianStatusCodes.GoodEntryInserted]);
|
||||
(await u.UpdateDataAsync("upd", HistoryUpdateMode.Update, [V(T(1), 1)], ConformanceData.Editor, Ct)).ShouldBe([HistorianStatusCodes.GoodEntryInserted]);
|
||||
(await B.TagExistsAsync("ins")).ShouldBeTrue();
|
||||
(await B.TagExistsAsync("upd")).ShouldBeTrue();
|
||||
}
|
||||
@@ -81,14 +81,14 @@ public abstract class UpdateConformance : ConformanceBase
|
||||
{
|
||||
var u = B.CreateUpdater();
|
||||
var r = await u.UpdateDataAsync("t", HistoryUpdateMode.Insert,
|
||||
[V(null, 1), V(T(1), 2), V(DateTime.MinValue, 3)], Ct);
|
||||
[V(null, 1), V(T(1), 2), V(DateTime.MinValue, 3)], ConformanceData.Editor, Ct);
|
||||
r.ShouldBe([HistorianStatusCodes.BadInvalidTimestamp, HistorianStatusCodes.GoodEntryInserted, HistorianStatusCodes.BadInvalidTimestamp]);
|
||||
(await B.ReadBackAsync("t")).Count.ShouldBe(1);
|
||||
|
||||
// All-invalid request: no DB work, no tag created, still one code per item.
|
||||
var u2 = B.CreateUpdater();
|
||||
await u2.UpdateDataAsync("seed", HistoryUpdateMode.Insert, [V(T(1), 1)], Ct); // bootstraps the schema
|
||||
(await u2.UpdateDataAsync("x", HistoryUpdateMode.Insert, [V(null, 1)], Ct)).ShouldBe([HistorianStatusCodes.BadInvalidTimestamp]);
|
||||
await u2.UpdateDataAsync("seed", HistoryUpdateMode.Insert, [V(T(1), 1)], ConformanceData.Editor, Ct); // bootstraps the schema
|
||||
(await u2.UpdateDataAsync("x", HistoryUpdateMode.Insert, [V(null, 1)], ConformanceData.Editor, Ct)).ShouldBe([HistorianStatusCodes.BadInvalidTimestamp]);
|
||||
(await B.TagExistsAsync("seed")).ShouldBeTrue();
|
||||
(await B.TagExistsAsync("x")).ShouldBeFalse(); // "x" was never created
|
||||
}
|
||||
@@ -98,7 +98,7 @@ public abstract class UpdateConformance : ConformanceBase
|
||||
{
|
||||
var u = B.CreateUpdater();
|
||||
var r = await u.UpdateDataAsync("t", HistoryUpdateMode.Insert,
|
||||
[V(T(1), double.NaN), V(T(2), 2), V(T(3), double.PositiveInfinity), V(T(4), double.NegativeInfinity)], Ct);
|
||||
[V(T(1), double.NaN), V(T(2), 2), V(T(3), double.PositiveInfinity), V(T(4), double.NegativeInfinity)], ConformanceData.Editor, Ct);
|
||||
r.ShouldBe([HistorianStatusCodes.BadInvalidArgument, HistorianStatusCodes.GoodEntryInserted,
|
||||
HistorianStatusCodes.BadInvalidArgument, HistorianStatusCodes.BadInvalidArgument]);
|
||||
(await B.ReadBackAsync("t")).Select(x => (double?)x.V.Number).ShouldBe([2d]);
|
||||
@@ -108,7 +108,7 @@ public abstract class UpdateConformance : ConformanceBase
|
||||
public async Task Duplicate_timestamp_in_one_Insert_request_second_is_BadEntryExists_first_wins()
|
||||
{
|
||||
var u = B.CreateUpdater();
|
||||
var r = await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T(1), 1), V(T(2), 2), V(T(1), 99)], Ct);
|
||||
var r = await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T(1), 1), V(T(2), 2), V(T(1), 99)], ConformanceData.Editor, Ct);
|
||||
r.ShouldBe([HistorianStatusCodes.GoodEntryInserted, HistorianStatusCodes.GoodEntryInserted, HistorianStatusCodes.BadEntryExists]);
|
||||
(await B.ReadBackAsync("t")).Select(x => (double?)x.V.Number).ShouldBe([1d, 2d]);
|
||||
}
|
||||
@@ -117,12 +117,12 @@ public abstract class UpdateConformance : ConformanceBase
|
||||
public async Task Duplicate_timestamp_in_one_Replace_or_Update_request_last_wins_and_all_share_the_code()
|
||||
{
|
||||
var u = B.CreateUpdater();
|
||||
await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T(1), 1)], Ct);
|
||||
var rep = await u.UpdateDataAsync("t", HistoryUpdateMode.Replace, [V(T(1), 5), V(T(1), 6)], Ct);
|
||||
await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T(1), 1)], ConformanceData.Editor, Ct);
|
||||
var rep = await u.UpdateDataAsync("t", HistoryUpdateMode.Replace, [V(T(1), 5), V(T(1), 6)], ConformanceData.Editor, Ct);
|
||||
rep.ShouldBe([HistorianStatusCodes.GoodEntryReplaced, HistorianStatusCodes.GoodEntryReplaced]);
|
||||
(await B.ReadBackAsync("t")).Single().V.Number.ShouldBe(6d);
|
||||
|
||||
var upd = await u.UpdateDataAsync("t", HistoryUpdateMode.Update, [V(T(7), 7), V(T(7), 8)], Ct);
|
||||
var upd = await u.UpdateDataAsync("t", HistoryUpdateMode.Update, [V(T(7), 7), V(T(7), 8)], ConformanceData.Editor, Ct);
|
||||
upd.ShouldBe([HistorianStatusCodes.GoodEntryInserted, HistorianStatusCodes.GoodEntryInserted]);
|
||||
(await B.ReadBackAsync("t")).Last().V.Number.ShouldBe(8d);
|
||||
}
|
||||
@@ -131,9 +131,9 @@ public abstract class UpdateConformance : ConformanceBase
|
||||
public async Task Replace_and_Update_can_change_the_value_kind_and_clear_stale_columns()
|
||||
{
|
||||
var u = B.CreateUpdater();
|
||||
await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [new HistorizationValue(T(1), HistorianValue.FromDouble(1.5), Good)], Ct);
|
||||
await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [new HistorizationValue(T(1), HistorianValue.FromDouble(1.5), Good)], ConformanceData.Editor, Ct);
|
||||
(await u.UpdateDataAsync("t", HistoryUpdateMode.Replace,
|
||||
[new HistorizationValue(T(1), HistorianValue.FromString("hi"), Good)], Ct)).ShouldBe([HistorianStatusCodes.GoodEntryReplaced]);
|
||||
[new HistorizationValue(T(1), HistorianValue.FromString("hi"), Good)], ConformanceData.Editor, Ct)).ShouldBe([HistorianStatusCodes.GoodEntryReplaced]);
|
||||
var rows = await B.ReadBackAsync("t");
|
||||
rows.Single().V.ShouldBe(HistorianValue.FromString("hi")); // Number cleared (0), not left at 1.5
|
||||
}
|
||||
@@ -142,13 +142,13 @@ public abstract class UpdateConformance : ConformanceBase
|
||||
public async Task DeleteRaw_is_half_open_Good_when_rows_removed_BadNoData_when_none()
|
||||
{
|
||||
var u = B.CreateUpdater();
|
||||
await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T(1), 1), V(T(2), 2), V(T(3), 3)], Ct);
|
||||
await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T(1), 1), V(T(2), 2), V(T(3), 3)], ConformanceData.Editor, Ct);
|
||||
// [T(1), T(3)) removes T(1) and T(2); T(3) (the end) survives.
|
||||
(await u.DeleteRawAsync("t", T(1), T(3), Ct)).ShouldBe(Good);
|
||||
(await u.DeleteRawAsync("t", T(1), T(3), ConformanceData.Editor, Ct)).ShouldBe(Good);
|
||||
(await B.ReadBackAsync("t")).Select(x => x.T).ShouldBe([B.Round(T(3))]);
|
||||
(await u.DeleteRawAsync("t", T(1), T(3), Ct)).ShouldBe(HistorianStatusCodes.BadNoData);
|
||||
(await u.DeleteRawAsync("t", T(1), T(3), ConformanceData.Editor, Ct)).ShouldBe(HistorianStatusCodes.BadNoData);
|
||||
// sub-ms boundary: a start one resolution unit after T(3) excludes it.
|
||||
(await u.DeleteRawAsync("t", T(3).AddTicks(B.ResolutionTicks), T(3).AddTicks(100 * B.ResolutionTicks), Ct)).ShouldBe(HistorianStatusCodes.BadNoData);
|
||||
(await u.DeleteRawAsync("t", T(3).AddTicks(B.ResolutionTicks), T(3).AddTicks(100 * B.ResolutionTicks), ConformanceData.Editor, Ct)).ShouldBe(HistorianStatusCodes.BadNoData);
|
||||
(await B.ReadBackAsync("t")).Count.ShouldBe(1);
|
||||
}
|
||||
|
||||
@@ -156,8 +156,8 @@ public abstract class UpdateConformance : ConformanceBase
|
||||
public async Task DeleteAtTime_reports_per_time_in_order_with_exact_tick_matching()
|
||||
{
|
||||
var u = B.CreateUpdater();
|
||||
await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T(1), 1), V(T(2), 2), V(T(3), 3)], Ct);
|
||||
var r = await u.DeleteAtTimeAsync("t", [T(3), T(2).AddTicks(B.ResolutionTicks) /* near miss */, T(1), DateTime.MinValue], Ct);
|
||||
await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T(1), 1), V(T(2), 2), V(T(3), 3)], ConformanceData.Editor, Ct);
|
||||
var r = await u.DeleteAtTimeAsync("t", [T(3), T(2).AddTicks(B.ResolutionTicks) /* near miss */, T(1), DateTime.MinValue], ConformanceData.Editor, Ct);
|
||||
r.ShouldBe([Good, HistorianStatusCodes.BadNoEntryExists, Good, HistorianStatusCodes.BadInvalidTimestamp]);
|
||||
(await B.ReadBackAsync("t")).Select(x => x.T).ShouldBe([B.Round(T(2))]);
|
||||
}
|
||||
@@ -166,8 +166,8 @@ public abstract class UpdateConformance : ConformanceBase
|
||||
public async Task DeleteAtTime_duplicate_time_first_reports_Good_later_BadNoEntryExists()
|
||||
{
|
||||
var u = B.CreateUpdater();
|
||||
await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T(1), 1)], Ct);
|
||||
(await u.DeleteAtTimeAsync("t", [T(1), T(1)], Ct)).ShouldBe([Good, HistorianStatusCodes.BadNoEntryExists]);
|
||||
await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T(1), 1)], ConformanceData.Editor, Ct);
|
||||
(await u.DeleteAtTimeAsync("t", [T(1), T(1)], ConformanceData.Editor, Ct)).ShouldBe([Good, HistorianStatusCodes.BadNoEntryExists]);
|
||||
(await B.ReadBackAsync("t")).ShouldBeEmpty();
|
||||
}
|
||||
}
|
||||
|
||||
@@ -58,7 +58,7 @@ public sealed class FocasStatusFlagsTests
|
||||
var client = new FakeFocasClient { Status = new FocasStatus(Auto: 1, Run: 3, Motion: 1, Mstb: 2, Emergency: 0, Alarm: 1, Edit: 0, TmMode: 4) };
|
||||
var drv = NewDriver(client);
|
||||
await drv.InitializeAsync("{}", CancellationToken.None);
|
||||
await WaitAsync(() => drv.GetDeviceState(Host)?.LastStatus is not null);
|
||||
await WaitAsync(() => drv.GetDeviceState(Host) is { LastFixedPollUtc: not null, FixedTreeCache: not null, LastStatus: not null });
|
||||
|
||||
string[] fields = ["Auto", "Run", "Motion", "Mstb", "Emergency", "Alarm", "Edit", "TmMode"];
|
||||
int[] expected = [1, 3, 1, 2, 0, 1, 0, 4];
|
||||
@@ -82,7 +82,8 @@ public sealed class FocasStatusFlagsTests
|
||||
var drv = NewDriver(client,
|
||||
new FocasTagDefinition("EStop", Host, "FIXED:Status/Emergency", FocasDataType.Int32, Writable: false));
|
||||
await drv.InitializeAsync("{}", CancellationToken.None);
|
||||
await WaitAsync(() => drv.GetDeviceState(Host)?.LastStatus is not null);
|
||||
// Wait for full readiness: an authored FIXED read needs the cache and the poll stamp too (#673).
|
||||
await WaitAsync(() => drv.GetDeviceState(Host) is { LastFixedPollUtc: not null, FixedTreeCache: not null, LastStatus: not null });
|
||||
|
||||
var snap = (await drv.ReadAsync(["EStop"], CancellationToken.None))[0];
|
||||
snap.StatusCode.ShouldBe(FocasStatusMapper.Good);
|
||||
@@ -97,7 +98,7 @@ public sealed class FocasStatusFlagsTests
|
||||
var client = new FakeFocasClient { Status = new FocasStatus(1, 0, 0, 0, 0, 0, 0, 0) };
|
||||
var drv = NewDriver(client);
|
||||
await drv.InitializeAsync("{}", CancellationToken.None);
|
||||
await WaitAsync(() => drv.GetDeviceState(Host)?.LastStatus is not null);
|
||||
await WaitAsync(() => drv.GetDeviceState(Host) is { LastFixedPollUtc: not null, FixedTreeCache: not null, LastStatus: not null });
|
||||
|
||||
client.Status = new FocasStatus(1, 3, 1, 0, 0, 0, 0, 0);
|
||||
await WaitAsync(() => drv.GetDeviceState(Host)?.LastStatus?.Run == 3);
|
||||
|
||||
+5
-3
@@ -52,7 +52,7 @@ public sealed class BootstrapTests(QuestDbFixture fx)
|
||||
var p = fx.UniquePrefix();
|
||||
var (c, boot, _) = Make(p);
|
||||
await boot.EnsureAsync(default);
|
||||
(await TablesAsync(c, p)).ShouldBe([p + "event", p + "sample", p + "tag"]);
|
||||
(await TablesAsync(c, p)).ShouldBe([p + "event", p + "sample", p + "sample_modified", p + "tag"]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
@@ -69,7 +69,7 @@ public sealed class BootstrapTests(QuestDbFixture fx)
|
||||
await Task.Delay(300); // an ALTER would have been applied by now
|
||||
(await SequencerTxnAsync(c2, p + "sample")).ShouldBe(before);
|
||||
(await SequencerTxnAsync(c2, p + "event")).ShouldBe(await SequencerTxnAsync(c, p + "event"));
|
||||
(await TablesAsync(c2, p)).Count.ShouldBe(3);
|
||||
(await TablesAsync(c2, p)).Count.ShouldBe(4);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
@@ -84,7 +84,7 @@ public sealed class BootstrapTests(QuestDbFixture fx)
|
||||
await Task.WhenAll(
|
||||
Task.Run(() => a.Boot.EnsureAsync(default)),
|
||||
Task.Run(() => b.Boot.EnsureAsync(default)));
|
||||
(await TablesAsync(a.Conn, p)).Count.ShouldBe(3);
|
||||
(await TablesAsync(a.Conn, p)).Count.ShouldBe(4);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -97,6 +97,8 @@ public sealed class BootstrapTests(QuestDbFixture fx)
|
||||
await b1.EnsureAsync(default);
|
||||
QuestDbTables.TtlHours((await TtlAsync(c1, withTtl + "sample")).Value, (await TtlAsync(c1, withTtl + "sample")).Unit).ShouldBe(168);
|
||||
QuestDbTables.TtlHours((await TtlAsync(c1, withTtl + "event")).Value, (await TtlAsync(c1, withTtl + "event")).Unit).ShouldBe(168);
|
||||
// #668: the modification trail ages out with the samples it describes.
|
||||
QuestDbTables.TtlHours((await TtlAsync(c1, withTtl + "sample_modified")).Value, (await TtlAsync(c1, withTtl + "sample_modified")).Unit).ShouldBe(168);
|
||||
(await TtlAsync(c1, withTtl + "tag")).Value.ShouldBe(0); // the tag catalog never expires
|
||||
|
||||
var noTtl = fx.UniquePrefix();
|
||||
|
||||
+7
@@ -53,3 +53,10 @@ public sealed class QuestDbTagNameConformance(QuestDbFixture fx) : TagNameConfor
|
||||
/// <inheritdoc/>
|
||||
protected override Task<IConformanceBackend> CreateBackendAsync() => QuestDbConformanceBackend.CreateAsync(fx);
|
||||
}
|
||||
|
||||
[Collection(QuestDbCollection.Name)]
|
||||
public sealed class QuestDbModifiedValueConformance(QuestDbFixture fx) : ModifiedValueConformance
|
||||
{
|
||||
/// <inheritdoc/>
|
||||
protected override Task<IConformanceBackend> CreateBackendAsync() => QuestDbConformanceBackend.CreateAsync(fx);
|
||||
}
|
||||
|
||||
+4
@@ -75,6 +75,10 @@ internal sealed class QuestDbConformanceBackend : IConformanceBackend
|
||||
public IHistorianDataSource CreateDataSource(TimeProvider? clock = null) =>
|
||||
new QuestDbHistorianDataSource(Conn, Boot, NewCatalog(), timeProvider: clock);
|
||||
|
||||
/// <inheritdoc/>
|
||||
public IHistorianDataSource CreateDataSource(int maxTimeWeightedRawRows) =>
|
||||
new QuestDbHistorianDataSource(Conn, Boot, NewCatalog(), maxTimeWeightedRawRows: maxTimeWeightedRawRows);
|
||||
|
||||
/// <inheritdoc/>
|
||||
/// <remarks>The T4 live-value writer over a fresh catalog (fresh positive cache) and this instance's WAL wait.</remarks>
|
||||
public IHistorianValueWriter CreateValueWriter(Func<DateTime>? utcNow = null) =>
|
||||
|
||||
+3
-3
@@ -90,8 +90,8 @@ public sealed class ReadTests(QuestDbFixture fx) : IAsyncLifetime
|
||||
max.Value.ShouldBe(50d);
|
||||
max.SourceTimestampUtc.ShouldBe(_b.Round(At(40)));
|
||||
// The extreme DateTime range: MinValue/MaxValue are nudged (never bound as ±infinity, which would match nothing).
|
||||
(await src.ReadProcessedAsync("A", DateTime.MinValue, DateTime.MaxValue, TimeSpan.Zero, HistoryAggregateType.Total, Ct))
|
||||
.Samples.Single().Value.ShouldBe(120d);
|
||||
(await src.ReadProcessedAsync("A", DateTime.MinValue, DateTime.MaxValue, TimeSpan.Zero, HistoryAggregateType.Count, Ct))
|
||||
.Samples.Single().Value.ShouldBe(4);
|
||||
// A pre-1970 window holds no data and is not an error.
|
||||
(await src.ReadProcessedAsync("A", start, new DateTime(1969, 1, 1, 0, 0, 0, DateTimeKind.Utc), TimeSpan.FromDays(3650),
|
||||
HistoryAggregateType.Count, Ct)).Samples.ShouldAllBe(x => x.StatusCode == NoDataCalc);
|
||||
@@ -247,7 +247,7 @@ public sealed class ReadTests(QuestDbFixture fx) : IAsyncLifetime
|
||||
await Should.ThrowAsync<HistorianSchemaMissingException>(() => src.ReadEventsAsync(null, At(0), At(5), 0, Ct));
|
||||
// Client-request errors are thrown before any I/O and are not backend failures.
|
||||
await Should.ThrowAsync<HistoryAggregateNotSupportedException>(() =>
|
||||
src.ReadProcessedAsync("A", At(0), At(5), TimeSpan.Zero, HistoryAggregateType.Interpolative, Ct));
|
||||
src.ReadProcessedAsync("A", At(0), At(5), TimeSpan.Zero, HistoryAggregateType.StandardDeviationPopulation, Ct));
|
||||
await Should.ThrowAsync<ArgumentOutOfRangeException>(() =>
|
||||
src.ReadProcessedAsync("A", At(0), At(100), TimeSpan.FromTicks(10), HistoryAggregateType.Count, Ct));
|
||||
|
||||
|
||||
+1
-1
@@ -26,7 +26,7 @@ public sealed partial class TombstoneTests
|
||||
];
|
||||
|
||||
(await writer.WriteLiveValuesAsync("r", batch, default)).ShouldBeTrue();
|
||||
(await rig.Updater.DeleteAtTimeAsync("r", [T0], default)).ShouldBe([HistorianStatusCodes.Good]);
|
||||
(await rig.Updater.DeleteAtTimeAsync("r", [T0], UpdateTests.Who, default)).ShouldBe([HistorianStatusCodes.Good]);
|
||||
(await writer.WriteLiveValuesAsync("r", batch, default)).ShouldBeTrue(); // the redelivery
|
||||
|
||||
(await UpdateTests.RowsAsync(rig, "r", liveOnly: true)).Select(r => r.Num).ShouldBe([2d]);
|
||||
|
||||
+6
-6
@@ -17,8 +17,8 @@ public sealed partial class TombstoneTests(QuestDbFixture fx)
|
||||
fx.SkipIfUnavailable();
|
||||
var rig = UpdateTests.MakeRig(fx);
|
||||
await rig.Updater.UpdateDataAsync("t", HistoryUpdateMode.Insert,
|
||||
[UpdateTests.V(T0, 1), UpdateTests.V(T0.AddSeconds(1), 2)], default);
|
||||
(await rig.Updater.DeleteAtTimeAsync("t", [T0], default)).ShouldBe([HistorianStatusCodes.Good]);
|
||||
[UpdateTests.V(T0, 1), UpdateTests.V(T0.AddSeconds(1), 2)], UpdateTests.Who, default);
|
||||
(await rig.Updater.DeleteAtTimeAsync("t", [T0], UpdateTests.Who, default)).ShouldBe([HistorianStatusCodes.Good]);
|
||||
|
||||
(await UpdateTests.RowsAsync(rig, "t", liveOnly: true)).Select(r => r.Num).ShouldBe([2d]);
|
||||
var all = await UpdateTests.RowsAsync(rig, "t");
|
||||
@@ -35,12 +35,12 @@ public sealed partial class TombstoneTests(QuestDbFixture fx)
|
||||
{
|
||||
fx.SkipIfUnavailable();
|
||||
var rig = UpdateTests.MakeRig(fx);
|
||||
await rig.Updater.UpdateDataAsync("t", HistoryUpdateMode.Insert, [UpdateTests.V(T0, 1)], default);
|
||||
await rig.Updater.DeleteAtTimeAsync("t", [T0], default);
|
||||
await rig.Updater.UpdateDataAsync("t", HistoryUpdateMode.Insert, [UpdateTests.V(T0, 1)], UpdateTests.Who, default);
|
||||
await rig.Updater.DeleteAtTimeAsync("t", [T0], UpdateTests.Who, default);
|
||||
|
||||
(await rig.Updater.UpdateDataAsync("t", HistoryUpdateMode.Replace, [UpdateTests.V(T0, 5)], default))
|
||||
(await rig.Updater.UpdateDataAsync("t", HistoryUpdateMode.Replace, [UpdateTests.V(T0, 5)], UpdateTests.Who, default))
|
||||
.ShouldBe([HistorianStatusCodes.BadNoEntryExists]); // a tombstone is no entry for Replace
|
||||
(await rig.Updater.UpdateDataAsync("t", HistoryUpdateMode.Insert, [UpdateTests.V(T0, 9)], default))
|
||||
(await rig.Updater.UpdateDataAsync("t", HistoryUpdateMode.Insert, [UpdateTests.V(T0, 9)], UpdateTests.Who, default))
|
||||
.ShouldBe([HistorianStatusCodes.GoodEntryInserted]);
|
||||
var live = await UpdateTests.RowsAsync(rig, "t", liveOnly: true);
|
||||
live.Count.ShouldBe(1);
|
||||
|
||||
+37
-34
@@ -24,6 +24,9 @@ public sealed class UpdateTests(QuestDbFixture fx)
|
||||
return new Rig(c, new QuestDbHistorianDataUpdater(c, tags, wal), tags);
|
||||
}
|
||||
|
||||
/// <summary>The HistoryUpdate caller these tests pass to the updater (recorded in the modification trail).</summary>
|
||||
internal static readonly HistoryUpdateContext Who = new("questdb-test", null);
|
||||
|
||||
internal static HistorizationValue V(DateTime? t, double v, uint status = Good) =>
|
||||
new(t, HistorianValue.FromDouble(v), status);
|
||||
|
||||
@@ -57,10 +60,10 @@ public sealed class UpdateTests(QuestDbFixture fx)
|
||||
{
|
||||
fx.SkipIfUnavailable();
|
||||
var rig = MakeRig(fx);
|
||||
(await rig.Updater.UpdateDataAsync("t1", HistoryUpdateMode.Insert, [V(T0, 1)], default))
|
||||
(await rig.Updater.UpdateDataAsync("t1", HistoryUpdateMode.Insert, [V(T0, 1)], UpdateTests.Who, default))
|
||||
.ShouldBe([HistorianStatusCodes.GoodEntryInserted]);
|
||||
var codes = await rig.Updater.UpdateDataAsync("t1", HistoryUpdateMode.Insert,
|
||||
[V(T0, 99), V(T0.AddSeconds(1), 2)], default);
|
||||
[V(T0, 99), V(T0.AddSeconds(1), 2)], UpdateTests.Who, default);
|
||||
codes.ShouldBe([HistorianStatusCodes.BadEntryExists, HistorianStatusCodes.GoodEntryInserted]);
|
||||
var rows = await RowsAsync(rig, "t1"); // read right after return: read-your-writes
|
||||
rows.Select(r => r.Num).ShouldBe([1d, 2d]);
|
||||
@@ -72,14 +75,14 @@ public sealed class UpdateTests(QuestDbFixture fx)
|
||||
{
|
||||
fx.SkipIfUnavailable();
|
||||
var rig = MakeRig(fx);
|
||||
(await rig.Updater.UpdateDataAsync("nope", HistoryUpdateMode.Replace, [V(T0, 1)], default))
|
||||
(await rig.Updater.UpdateDataAsync("nope", HistoryUpdateMode.Replace, [V(T0, 1)], UpdateTests.Who, default))
|
||||
.ShouldBe([HistorianStatusCodes.BadNoEntryExists]);
|
||||
(await TagRowsAsync(rig, "nope")).ShouldBe(0);
|
||||
(await RowsAsync(rig, "nope")).ShouldBeEmpty();
|
||||
|
||||
await rig.Updater.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T0, 1)], default);
|
||||
await rig.Updater.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T0, 1)], UpdateTests.Who, default);
|
||||
var codes = await rig.Updater.UpdateDataAsync("t", HistoryUpdateMode.Replace,
|
||||
[V(T0, 5), V(T0.AddSeconds(1), 6)], default);
|
||||
[V(T0, 5), V(T0.AddSeconds(1), 6)], UpdateTests.Who, default);
|
||||
codes.ShouldBe([HistorianStatusCodes.GoodEntryReplaced, HistorianStatusCodes.BadNoEntryExists]);
|
||||
(await RowsAsync(rig, "t")).Select(r => r.Num).ShouldBe([5d]);
|
||||
}
|
||||
@@ -89,9 +92,9 @@ public sealed class UpdateTests(QuestDbFixture fx)
|
||||
{
|
||||
fx.SkipIfUnavailable();
|
||||
var rig = MakeRig(fx);
|
||||
await rig.Updater.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T0, 1)], default);
|
||||
await rig.Updater.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T0, 1)], UpdateTests.Who, default);
|
||||
var codes = await rig.Updater.UpdateDataAsync("t", HistoryUpdateMode.Update,
|
||||
[V(T0, 2), V(T0.AddSeconds(1), 3), V(T0.AddSeconds(1).AddTicks(3), 4)], default);
|
||||
[V(T0, 2), V(T0.AddSeconds(1), 3), V(T0.AddSeconds(1).AddTicks(3), 4)], UpdateTests.Who, default);
|
||||
codes.ShouldBe([HistorianStatusCodes.GoodEntryReplaced, HistorianStatusCodes.GoodEntryInserted,
|
||||
HistorianStatusCodes.GoodEntryInserted]);
|
||||
(await RowsAsync(rig, "t")).Select(r => r.Num).ShouldBe([2d, 4d]);
|
||||
@@ -104,7 +107,7 @@ public sealed class UpdateTests(QuestDbFixture fx)
|
||||
var rig = MakeRig(fx);
|
||||
// offsets 0 and 3 ticks sit inside one 10-tick window => the same stored µs
|
||||
var codes = await rig.Updater.UpdateDataAsync("t", HistoryUpdateMode.Insert,
|
||||
[V(T0, 1), V(T0.AddTicks(3), 2)], default);
|
||||
[V(T0, 1), V(T0.AddTicks(3), 2)], UpdateTests.Who, default);
|
||||
codes.ShouldBe([HistorianStatusCodes.GoodEntryInserted, HistorianStatusCodes.BadEntryExists]);
|
||||
(await RowsAsync(rig, "t")).Select(r => r.Num).ShouldBe([1d]);
|
||||
}
|
||||
@@ -119,7 +122,7 @@ public sealed class UpdateTests(QuestDbFixture fx)
|
||||
V(null, 1), V(DateTime.MinValue, 1), V(T0, double.NaN),
|
||||
new HistorizationValue(T0.AddSeconds(1), HistorianValue.Empty, Good),
|
||||
V(T0.AddSeconds(2), 7),
|
||||
], default);
|
||||
], UpdateTests.Who, default);
|
||||
codes.ShouldBe([HistorianStatusCodes.BadInvalidTimestamp, HistorianStatusCodes.BadInvalidTimestamp,
|
||||
HistorianStatusCodes.BadInvalidArgument, HistorianStatusCodes.BadInvalidArgument,
|
||||
HistorianStatusCodes.GoodEntryInserted]);
|
||||
@@ -132,10 +135,10 @@ public sealed class UpdateTests(QuestDbFixture fx)
|
||||
fx.SkipIfUnavailable();
|
||||
var rig = MakeRig(fx);
|
||||
var tag = new string('x', HistorianTagNames.MaxLength + 1);
|
||||
(await rig.Updater.UpdateDataAsync(tag, HistoryUpdateMode.Insert, [V(T0, 1)], default))
|
||||
(await rig.Updater.UpdateDataAsync(tag, HistoryUpdateMode.Insert, [V(T0, 1)], UpdateTests.Who, default))
|
||||
.ShouldBe([HistorianStatusCodes.BadInvalidArgument]);
|
||||
(await rig.Updater.DeleteAtTimeAsync(tag, [T0], default)).ShouldBe([HistorianStatusCodes.BadInvalidArgument]);
|
||||
(await rig.Updater.DeleteRawAsync(tag, T0, T0.AddHours(1), default)).ShouldBe(HistorianStatusCodes.BadInvalidArgument);
|
||||
(await rig.Updater.DeleteAtTimeAsync(tag, [T0], UpdateTests.Who, default)).ShouldBe([HistorianStatusCodes.BadInvalidArgument]);
|
||||
(await rig.Updater.DeleteRawAsync(tag, T0, T0.AddHours(1), UpdateTests.Who, default)).ShouldBe(HistorianStatusCodes.BadInvalidArgument);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
@@ -146,15 +149,15 @@ public sealed class UpdateTests(QuestDbFixture fx)
|
||||
var pre = new DateTime(1969, 12, 31, 23, 59, 59, DateTimeKind.Utc);
|
||||
foreach (var mode in new[] { HistoryUpdateMode.Insert, HistoryUpdateMode.Replace, HistoryUpdateMode.Update })
|
||||
{
|
||||
var codes = await rig.Updater.UpdateDataAsync("t", mode, [V(pre, 1), V(DateTime.MinValue, 1), V(T0, 2)], default);
|
||||
var codes = await rig.Updater.UpdateDataAsync("t", mode, [V(pre, 1), V(DateTime.MinValue, 1), V(T0, 2)], UpdateTests.Who, default);
|
||||
codes[0].ShouldBe(HistorianStatusCodes.BadOutOfRange);
|
||||
codes[1].ShouldBe(HistorianStatusCodes.BadInvalidTimestamp);
|
||||
}
|
||||
// Pre-1970 only: the whole batch is refused, no tag row is created, no INSERT would succeed anyway.
|
||||
(await rig.Updater.UpdateDataAsync("only-pre", HistoryUpdateMode.Update, [V(pre, 1)], default))
|
||||
(await rig.Updater.UpdateDataAsync("only-pre", HistoryUpdateMode.Update, [V(pre, 1)], UpdateTests.Who, default))
|
||||
.ShouldBe([HistorianStatusCodes.BadOutOfRange]);
|
||||
(await TagRowsAsync(rig, "only-pre")).ShouldBe(0);
|
||||
(await rig.Updater.DeleteAtTimeAsync("t", [pre], default)).ShouldBe([HistorianStatusCodes.BadNoEntryExists]);
|
||||
(await rig.Updater.DeleteAtTimeAsync("t", [pre], UpdateTests.Who, default)).ShouldBe([HistorianStatusCodes.BadNoEntryExists]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
@@ -162,15 +165,15 @@ public sealed class UpdateTests(QuestDbFixture fx)
|
||||
{
|
||||
fx.SkipIfUnavailable();
|
||||
var rig = MakeRig(fx);
|
||||
await rig.Updater.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T0, 1), V(T0.AddSeconds(1), 2)], default);
|
||||
await rig.Updater.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T0, 1), V(T0.AddSeconds(1), 2)], UpdateTests.Who, default);
|
||||
var codes = await rig.Updater.DeleteAtTimeAsync("t",
|
||||
[T0, T0.AddSeconds(5), T0, DateTime.MinValue], default);
|
||||
[T0, T0.AddSeconds(5), T0, DateTime.MinValue], UpdateTests.Who, default);
|
||||
codes.ShouldBe([Good, HistorianStatusCodes.BadNoEntryExists, HistorianStatusCodes.BadNoEntryExists,
|
||||
HistorianStatusCodes.BadInvalidTimestamp]);
|
||||
(await RowsAsync(rig, "t", liveOnly: true)).Select(r => r.Num).ShouldBe([2d]);
|
||||
// a tombstone is not a live row: deleting it again finds nothing
|
||||
(await rig.Updater.DeleteAtTimeAsync("t", [T0], default)).ShouldBe([HistorianStatusCodes.BadNoEntryExists]);
|
||||
(await rig.Updater.DeleteAtTimeAsync("unknown", [T0], default)).ShouldBe([HistorianStatusCodes.BadNoEntryExists]);
|
||||
(await rig.Updater.DeleteAtTimeAsync("t", [T0], UpdateTests.Who, default)).ShouldBe([HistorianStatusCodes.BadNoEntryExists]);
|
||||
(await rig.Updater.DeleteAtTimeAsync("unknown", [T0], UpdateTests.Who, default)).ShouldBe([HistorianStatusCodes.BadNoEntryExists]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
@@ -178,9 +181,9 @@ public sealed class UpdateTests(QuestDbFixture fx)
|
||||
{
|
||||
fx.SkipIfUnavailable();
|
||||
var rig = MakeRig(fx);
|
||||
await rig.Updater.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T0, 1)], default);
|
||||
await rig.Updater.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T0, 1)], UpdateTests.Who, default);
|
||||
// +7 ticks is inside the same µs; the row lives at T0.
|
||||
(await rig.Updater.DeleteAtTimeAsync("t", [T0.AddTicks(7)], default)).ShouldBe([Good]);
|
||||
(await rig.Updater.DeleteAtTimeAsync("t", [T0.AddTicks(7)], UpdateTests.Who, default)).ShouldBe([Good]);
|
||||
(await RowsAsync(rig, "t", liveOnly: true)).ShouldBeEmpty();
|
||||
}
|
||||
|
||||
@@ -189,18 +192,18 @@ public sealed class UpdateTests(QuestDbFixture fx)
|
||||
{
|
||||
fx.SkipIfUnavailable();
|
||||
var rig = MakeRig(fx);
|
||||
(await rig.Updater.DeleteRawAsync("unknown", T0, T0.AddHours(1), default)).ShouldBe(HistorianStatusCodes.BadNoData);
|
||||
(await rig.Updater.DeleteRawAsync("unknown", T0, T0.AddHours(1), UpdateTests.Who, default)).ShouldBe(HistorianStatusCodes.BadNoData);
|
||||
|
||||
// 2,500 rows => 3 pages of at most 1,000; plus one row exactly at the (exclusive) end.
|
||||
var values = Enumerable.Range(0, 2500).Select(i => V(T0.AddSeconds(i), i)).ToList();
|
||||
values.Add(V(T0.AddSeconds(2500), -1));
|
||||
(await rig.Updater.UpdateDataAsync("big", HistoryUpdateMode.Insert, values, default))
|
||||
(await rig.Updater.UpdateDataAsync("big", HistoryUpdateMode.Insert, values, UpdateTests.Who, default))
|
||||
.ShouldAllBe(c => c == HistorianStatusCodes.GoodEntryInserted);
|
||||
(await rig.Updater.DeleteRawAsync("big", T0, T0.AddSeconds(2500), default)).ShouldBe(Good);
|
||||
(await rig.Updater.DeleteRawAsync("big", T0, T0.AddSeconds(2500), UpdateTests.Who, default)).ShouldBe(Good);
|
||||
var live = await RowsAsync(rig, "big", liveOnly: true);
|
||||
live.Count.ShouldBe(1);
|
||||
live[0].Num.ShouldBe(-1d);
|
||||
(await rig.Updater.DeleteRawAsync("big", T0, T0.AddSeconds(2500), default)).ShouldBe(HistorianStatusCodes.BadNoData);
|
||||
(await rig.Updater.DeleteRawAsync("big", T0, T0.AddSeconds(2500), UpdateTests.Who, default)).ShouldBe(HistorianStatusCodes.BadNoData);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
@@ -208,12 +211,12 @@ public sealed class UpdateTests(QuestDbFixture fx)
|
||||
{
|
||||
fx.SkipIfUnavailable();
|
||||
var rig = MakeRig(fx);
|
||||
await rig.Updater.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T0, 1), V(T0.AddSeconds(1), 2)], default);
|
||||
(await rig.Updater.DeleteRawAsync("t", new DateTime(1900, 1, 1, 0, 0, 0, DateTimeKind.Utc), T0.AddSeconds(1), default))
|
||||
await rig.Updater.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T0, 1), V(T0.AddSeconds(1), 2)], UpdateTests.Who, default);
|
||||
(await rig.Updater.DeleteRawAsync("t", new DateTime(1900, 1, 1, 0, 0, 0, DateTimeKind.Utc), T0.AddSeconds(1), UpdateTests.Who, default))
|
||||
.ShouldBe(Good);
|
||||
(await RowsAsync(rig, "t", liveOnly: true)).Select(r => r.Num).ShouldBe([2d]);
|
||||
// DateTime.MinValue as the start is nudged inside, not bound as -infinity
|
||||
(await rig.Updater.DeleteRawAsync("t", DateTime.MinValue, DateTime.MaxValue, default)).ShouldBe(Good);
|
||||
(await rig.Updater.DeleteRawAsync("t", DateTime.MinValue, DateTime.MaxValue, UpdateTests.Who, default)).ShouldBe(Good);
|
||||
(await RowsAsync(rig, "t", liveOnly: true)).ShouldBeEmpty();
|
||||
}
|
||||
|
||||
@@ -226,9 +229,9 @@ public sealed class UpdateTests(QuestDbFixture fx)
|
||||
for (var i = 0; i < 40; i++)
|
||||
{
|
||||
var t = T0.AddSeconds(i);
|
||||
(await rig.Updater.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(t, i)], default))
|
||||
(await rig.Updater.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(t, i)], UpdateTests.Who, default))
|
||||
.ShouldBe([HistorianStatusCodes.GoodEntryInserted]);
|
||||
(await rig.Updater.UpdateDataAsync("t", HistoryUpdateMode.Replace, [V(t, i + 1000)], default))
|
||||
(await rig.Updater.UpdateDataAsync("t", HistoryUpdateMode.Replace, [V(t, i + 1000)], UpdateTests.Who, default))
|
||||
.ShouldBe([HistorianStatusCodes.GoodEntryReplaced], $"iteration {i}: the insert was not visible");
|
||||
}
|
||||
}
|
||||
@@ -242,8 +245,8 @@ public sealed class UpdateTests(QuestDbFixture fx)
|
||||
{
|
||||
var t = T0.AddSeconds(i);
|
||||
var codes = await Task.WhenAll(
|
||||
Task.Run(() => rig.Updater.UpdateDataAsync("race", HistoryUpdateMode.Insert, [V(t, 1)], default)),
|
||||
Task.Run(() => rig.Updater.UpdateDataAsync("race", HistoryUpdateMode.Insert, [V(t, 2)], default)));
|
||||
Task.Run(() => rig.Updater.UpdateDataAsync("race", HistoryUpdateMode.Insert, [V(t, 1)], UpdateTests.Who, default)),
|
||||
Task.Run(() => rig.Updater.UpdateDataAsync("race", HistoryUpdateMode.Insert, [V(t, 2)], UpdateTests.Who, default)));
|
||||
codes.Select(c => c[0]).OrderBy(c => c).ToArray().ShouldBe(
|
||||
new[] { HistorianStatusCodes.GoodEntryInserted, HistorianStatusCodes.BadEntryExists }, $"iteration {i}");
|
||||
}
|
||||
@@ -256,10 +259,10 @@ public sealed class UpdateTests(QuestDbFixture fx)
|
||||
var rig = MakeRig(fx);
|
||||
// 1 Hz seed across 2 h = 7,200 live rows; the request touches 3 instants 1 h apart.
|
||||
var seed = Enumerable.Range(0, 7200).Select(i => V(T0.AddSeconds(i), i)).ToList();
|
||||
await rig.Updater.UpdateDataAsync("sparse", HistoryUpdateMode.Insert, seed, default);
|
||||
await rig.Updater.UpdateDataAsync("sparse", HistoryUpdateMode.Insert, seed, UpdateTests.Who, default);
|
||||
var before = rig.Updater.LiveRowsRead;
|
||||
var codes = await rig.Updater.UpdateDataAsync("sparse", HistoryUpdateMode.Replace,
|
||||
[V(T0, 1), V(T0.AddHours(1), 2), V(T0.AddHours(2).AddSeconds(-1), 3), V(T0.AddHours(1).AddMilliseconds(500), 4)], default);
|
||||
[V(T0, 1), V(T0.AddHours(1), 2), V(T0.AddHours(2).AddSeconds(-1), 3), V(T0.AddHours(1).AddMilliseconds(500), 4)], UpdateTests.Who, default);
|
||||
codes.ShouldBe([HistorianStatusCodes.GoodEntryReplaced, HistorianStatusCodes.GoodEntryReplaced,
|
||||
HistorianStatusCodes.GoodEntryReplaced, HistorianStatusCodes.BadNoEntryExists]);
|
||||
// 3 clusters of a single instant each (T0+1h and +1h.5s are one cluster of 2 instants, 1 live row)
|
||||
|
||||
+13
-1
@@ -39,10 +39,22 @@ public sealed class QuestDbTablesTests
|
||||
[Fact]
|
||||
public void The_prefix_is_applied_to_every_table_name()
|
||||
{
|
||||
QuestDbTables.All("t1_").ShouldBe(["t1_sample", "t1_event", "t1_tag"]);
|
||||
QuestDbTables.All("t1_").ShouldBe(["t1_sample", "t1_event", "t1_tag", "t1_sample_modified"]);
|
||||
QuestDbTables.Expiring("t1_").ShouldBe(["t1_sample", "t1_event", "t1_sample_modified"]);
|
||||
QuestDbTables.CreateAll("t1_", 3).ShouldAllBe(s => s.Contains("t1_"));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Sample_modified_ddl_is_pinned_append_only_and_carries_the_ttl()
|
||||
{
|
||||
QuestDbTables.CreateSampleModified("hist_", null).ShouldBe(
|
||||
"CREATE TABLE IF NOT EXISTS hist_sample_modified (ts TIMESTAMP, tag SYMBOL, modified_id LONG, " +
|
||||
"modification_ts TIMESTAMP, update_type BYTE, principal VARCHAR, api_key_id VARCHAR, kind BYTE, num_value DOUBLE, " +
|
||||
"int_value LONG, str_value VARCHAR, status LONG, server_ts TIMESTAMP) TIMESTAMP(ts) PARTITION BY DAY WAL");
|
||||
QuestDbTables.CreateSampleModified("hist_", 30).ShouldEndWith("PARTITION BY DAY TTL 30 DAYS WAL");
|
||||
QuestDbTables.CreateSampleModified("hist_", null).ShouldNotContain("DEDUP");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void An_invalid_prefix_is_refused() =>
|
||||
Should.Throw<ArgumentException>(() => QuestDbTables.CreateSample("a b", null));
|
||||
|
||||
-1
@@ -46,7 +46,6 @@ public sealed class AggregateEvaluatorTests
|
||||
[InlineData(HistoryAggregateType.Average, 15d, 40d, 50d)]
|
||||
[InlineData(HistoryAggregateType.Minimum, 10d, 40d, 50d)]
|
||||
[InlineData(HistoryAggregateType.Maximum, 20d, 40d, 50d)]
|
||||
[InlineData(HistoryAggregateType.Total, 30d, 40d, 50d)]
|
||||
[InlineData(HistoryAggregateType.Start, 10d, 40d, 50d)]
|
||||
[InlineData(HistoryAggregateType.End, 20d, 40d, 50d)]
|
||||
[InlineData(HistoryAggregateType.Delta, 10d, 0d, 0d)]
|
||||
|
||||
+6
-2
@@ -10,7 +10,11 @@ public sealed class ProcessedReadTests
|
||||
[Fact]
|
||||
public void Every_enum_member_is_either_served_or_refused()
|
||||
{
|
||||
// Thirteen served; anything appended to the enum later falls to the refusal (total default).
|
||||
Enum.GetValues<HistoryAggregateType>().Count(a => AggregateFamilies.FamilyOf(a) is not null).ShouldBe(13);
|
||||
// Seventeen served (#671 added TimeAverage, TimeAverage2, Interpolative and Total2, and moved Total to the
|
||||
// time-weighted family); anything appended to the enum later falls to the refusal (total default).
|
||||
Enum.GetValues<HistoryAggregateType>().Count(a => AggregateFamilies.FamilyOf(a) is not null).ShouldBe(17);
|
||||
Enum.GetValues<HistoryAggregateType>().Where(a => AggregateFamilies.FamilyOf(a) == AggregateFamily.TimeWeighted)
|
||||
.ShouldBe([HistoryAggregateType.Total, HistoryAggregateType.Interpolative, HistoryAggregateType.TimeAverage,
|
||||
HistoryAggregateType.TimeAverage2, HistoryAggregateType.Total2]);
|
||||
}
|
||||
}
|
||||
|
||||
+8
@@ -39,6 +39,14 @@ public sealed class StatusCodeOracleTests
|
||||
HistorianInfoBits.WithCalculated(interpolatedGood).ShouldBe(0x400u | HistorianInfoBits.HistorianCalculated);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Partial_bit_matches_the_SDK() =>
|
||||
((uint)Opc.Ua.AggregateBits.Partial).ShouldBe(HistorianInfoBits.HistorianPartial);
|
||||
|
||||
[Fact]
|
||||
public void Generic_bad_matches_the_SDK() =>
|
||||
HistorianStatusCodes.Bad.ShouldBe(Opc.Ua.StatusCodes.Bad);
|
||||
|
||||
[Fact]
|
||||
public void Info_type_is_data_value() =>
|
||||
HistorianInfoBits.InfoTypeDataValue.ShouldBe(0x400u);
|
||||
|
||||
+430
@@ -0,0 +1,430 @@
|
||||
using Shouldly;
|
||||
using Xunit;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
|
||||
using static ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests.Fixture;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// The Part 13 time-weighted aggregates (#671) over hand-built sample streams. Every literal is hand-computed in the
|
||||
/// comment above it (areas in value·seconds; "raw" = the simple-bounding signal, "bridged" = interpolated bounding).
|
||||
/// </summary>
|
||||
public sealed class TimeWeightedAggregatorTests
|
||||
{
|
||||
private static readonly DateTime Now = At(3600);
|
||||
private const uint GoodRaw = 0x00000000u;
|
||||
private const uint NoData = 0x809B0000u;
|
||||
private const uint InvalidInputs = 0x80D60000u;
|
||||
private const uint BadCalc = 0x80000401u; // generic Bad | Calculated (an all-outage interval)
|
||||
private const uint GoodCalcPartial = 0x00000405u; // Good | Calculated | Partial (truncated last interval)
|
||||
|
||||
private static IReadOnlyList<DataValueSnapshot> Run(HistoryAggregateType agg, double from, double to, double intervalS,
|
||||
params StoredSample[] samples) =>
|
||||
TimeWeightedAggregator.Evaluate(agg, BucketPlan.Create(At(from), At(to), TimeSpan.FromSeconds(intervalS)), samples, Now);
|
||||
|
||||
private static void Values(IReadOnlyList<DataValueSnapshot> s, params double?[] expected)
|
||||
{
|
||||
s.Count.ShouldBe(expected.Length);
|
||||
for (var i = 0; i < expected.Length; i++)
|
||||
{
|
||||
if (expected[i] is { } e) ((double)s[i].Value!).ShouldBe(e, 1e-9, $"bucket {i}");
|
||||
else s[i].Value.ShouldBeNull($"bucket {i}");
|
||||
}
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------- tag A: +0=10 G, +10=20 G, +20=Bad, +30=40 G, +40=50 G
|
||||
// Window [0,80), 20 s buckets. Raw: [0,10) 10→20 Good (150); [10,20) 20 held into the outage, Uncertain (200);
|
||||
// [20,30) Bad; [30,40) 40→50 Good (450); [40,80) 50 extrapolated, Uncertain (2000).
|
||||
// Bridged: [0,10) 150; [10,30) 20→40 = 250 in B0 + 350 in B1; [30,40) 450; [40,80) 50 held.
|
||||
|
||||
[Fact]
|
||||
public void TimeAverage_bridges_the_outage_on_tag_A()
|
||||
{
|
||||
// B0 (150+250)/20 = 20; B1 (350+450)/20 = 40; B2, B3 = 50. Status (SDK rule, Uncertain time counts as Good):
|
||||
// B0 is non-Bad throughout (held-into-outage time is Uncertain) ⇒ Good; B1 has the Bad [20,30) ⇒ Uncertain;
|
||||
// B2, B3 are extrapolated (Uncertain) throughout ⇒ Good.
|
||||
var s = Run(HistoryAggregateType.TimeAverage, 0, 80, 20, A);
|
||||
Values(s, 20, 40, 50, 50);
|
||||
s.Select(x => x.StatusCode).ShouldBe([GoodCalc, UncertainCalc, GoodCalc, GoodCalc]);
|
||||
s.Select(x => x.SourceTimestampUtc).ShouldBe([At(0), At(20), At(40), At(60)]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Total_is_TimeAverage_times_interval_seconds_on_tag_A()
|
||||
{
|
||||
// 20×20, 40×20, 50×20, 50×20.
|
||||
Values(Run(HistoryAggregateType.Total, 0, 80, 20, A), 400, 800, 1000, 1000);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void TimeAverage2_excludes_the_bad_interval_on_tag_A()
|
||||
{
|
||||
// B0 350/20 = 17.5; B1 450/10 = 45 (the Bad [20,30) excluded); B2, B3 = 50.
|
||||
var s = Run(HistoryAggregateType.TimeAverage2, 0, 80, 20, A);
|
||||
Values(s, 17.5, 45, 50, 50);
|
||||
s.Select(x => x.StatusCode).ShouldBe([GoodCalc, UncertainCalc, GoodCalc, GoodCalc]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Total2_integrates_only_the_non_bad_data_on_tag_A()
|
||||
{
|
||||
// B0 150+200 = 350; B1 450; B2, B3 1000.
|
||||
Values(Run(HistoryAggregateType.Total2, 0, 80, 20, A), 350, 450, 1000, 1000);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Interpolative_on_tag_A()
|
||||
{
|
||||
// +0 exact Good 10 (raw status); +20 sits on the Bad sample ⇒ bridged 20→40 at the midpoint = 30, Uncertain;
|
||||
// +40 exact Good 50; +60 past the last sample ⇒ 50 extrapolated, Uncertain.
|
||||
var s = Run(HistoryAggregateType.Interpolative, 0, 80, 20, A);
|
||||
Values(s, 10, 30, 50, 50);
|
||||
s.Select(x => x.StatusCode).ShouldBe([GoodRaw, UncertainInterpolated, GoodRaw, UncertainInterpolated]);
|
||||
s.Select(x => x.SourceTimestampUtc).ShouldBe([At(0), At(20), At(40), At(60)]);
|
||||
s.ShouldAllBe(x => x.ServerTimestampUtc == Now);
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------- bounding values on both sides, crossing segments
|
||||
|
||||
private static readonly StoredSample[] Bounded =
|
||||
[
|
||||
S(-10, HistorianValue.FromDouble(0), Good), // bound before the window
|
||||
S(10, HistorianValue.FromDouble(20), Good),
|
||||
S(30, HistorianValue.FromDouble(20), Good), // bound after the window
|
||||
];
|
||||
|
||||
[Theory]
|
||||
// Window [0,20), 10 s. Segment -10..10 (0→20) crosses the window start: B0 [0,10) = 10→20, mean 15.
|
||||
// B1 [10,20) = 20 flat (crossing the window end into the bound). All Good.
|
||||
[InlineData(HistoryAggregateType.TimeAverage, 15d, 20d)]
|
||||
[InlineData(HistoryAggregateType.TimeAverage2, 15d, 20d)]
|
||||
[InlineData(HistoryAggregateType.Total, 150d, 200d)]
|
||||
[InlineData(HistoryAggregateType.Total2, 150d, 200d)]
|
||||
public void Bounding_values_before_and_after_the_window_are_used(HistoryAggregateType agg, double b0, double b1)
|
||||
{
|
||||
var s = Run(agg, 0, 20, 10, Bounded);
|
||||
Values(s, b0, b1);
|
||||
s.ShouldAllBe(x => x.StatusCode == GoodCalc);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Interpolative_interpolates_from_the_bound_before_the_window()
|
||||
{
|
||||
// +0: 0 + (20−0)·(10/20) = 10, Good both ends; +10 exact Good 20.
|
||||
var s = Run(HistoryAggregateType.Interpolative, 0, 20, 10, Bounded);
|
||||
Values(s, 10, 20);
|
||||
s.Select(x => x.StatusCode).ShouldBe([GoodInterpolated, GoodRaw]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void A_segment_spanning_several_buckets_is_split_at_every_edge()
|
||||
{
|
||||
// One segment 0..40 (0→40), window [0,40), 10 s: bucket means 5, 15, 25, 35; Total2 50, 150, 250, 350.
|
||||
StoredSample[] line = [S(0, HistorianValue.FromDouble(0), Good), S(40, HistorianValue.FromDouble(40), Good)];
|
||||
Values(Run(HistoryAggregateType.TimeAverage2, 0, 40, 10, line), 5, 15, 25, 35);
|
||||
Values(Run(HistoryAggregateType.Total2, 0, 40, 10, line), 50, 150, 250, 350);
|
||||
Values(Run(HistoryAggregateType.Interpolative, 0, 40, 10, line), 0, 10, 20, 30);
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------- empty buckets between bounds
|
||||
|
||||
[Fact]
|
||||
public void An_empty_bucket_with_bounds_on_both_sides_is_interpolated_not_no_data()
|
||||
{
|
||||
// -5 = 0, +25 = 30 (slope 1/s); window [0,20), 10 s holds no sample at all.
|
||||
StoredSample[] sparse = [S(-5, HistorianValue.FromDouble(0), Good), S(25, HistorianValue.FromDouble(30), Good)];
|
||||
var interp = Run(HistoryAggregateType.Interpolative, 0, 20, 10, sparse);
|
||||
Values(interp, 5, 15);
|
||||
interp.ShouldAllBe(x => x.StatusCode == GoodInterpolated);
|
||||
// B0 5→15 mean 10; B1 15→25 mean 20.
|
||||
var avg = Run(HistoryAggregateType.TimeAverage, 0, 20, 10, sparse);
|
||||
Values(avg, 10, 20);
|
||||
avg.ShouldAllBe(x => x.StatusCode == GoodCalc);
|
||||
Values(Run(HistoryAggregateType.Total, 0, 20, 10, sparse), 100, 200);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Past_the_last_sample_the_value_is_extrapolated_stepped()
|
||||
{
|
||||
// -5 = 7 and nothing later: held at 7 over [0,10). The extrapolated time is Uncertain, which the SDK rule
|
||||
// (TreatUncertainAsBad = false) counts as Good time ⇒ the interval is Good; the Interpolative point is Uncertain.
|
||||
StoredSample[] one = [S(-5, HistorianValue.FromDouble(7), Good)];
|
||||
var s = Run(HistoryAggregateType.TimeAverage2, 0, 10, 0, one).Single();
|
||||
((double)s.Value!).ShouldBe(7d);
|
||||
s.StatusCode.ShouldBe(GoodCalc);
|
||||
var i = Run(HistoryAggregateType.Interpolative, 0, 10, 0, one).Single();
|
||||
((double)i.Value!).ShouldBe(7d);
|
||||
i.StatusCode.ShouldBe(UncertainInterpolated);
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData(HistoryAggregateType.TimeAverage)]
|
||||
[InlineData(HistoryAggregateType.TimeAverage2)]
|
||||
[InlineData(HistoryAggregateType.Total)]
|
||||
[InlineData(HistoryAggregateType.Total2)]
|
||||
public void No_data_at_all_is_BadNoData_with_the_calculated_bit(HistoryAggregateType agg)
|
||||
{
|
||||
var s = Run(agg, 0, 20, 10);
|
||||
s.ShouldAllBe(x => x.StatusCode == NoDataCalc && x.Value == null);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Interpolative_with_no_data_is_plain_BadNoData()
|
||||
{
|
||||
// The only sample is at +15: nothing at or before either bucket start (+0, +10).
|
||||
var s = Run(HistoryAggregateType.Interpolative, 0, 20, 10, S(15, HistorianValue.FromDouble(1), Good));
|
||||
s.ShouldAllBe(x => x.StatusCode == NoData && x.Value == null);
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------- Bad intervals: TimeAverage vs TimeAverage2/Total2
|
||||
|
||||
private static readonly StoredSample[] Outage =
|
||||
[
|
||||
S(0, HistorianValue.FromDouble(10), Good),
|
||||
S(5, HistorianValue.Empty, BadNoCommunication),
|
||||
S(15, HistorianValue.FromDouble(30), Good),
|
||||
S(25, HistorianValue.FromDouble(30), Good), // bound after the window
|
||||
];
|
||||
|
||||
[Fact]
|
||||
public void TimeAverage2_and_Total2_exclude_the_bad_interval()
|
||||
{
|
||||
// Window [0,20) one bucket. Raw: [0,5) 10 held into the outage (50, Uncertain); [5,15) Bad; [15,20) 30 Good (150).
|
||||
// TimeAverage2 = 200 / 10 s = 20; Total2 = 200.
|
||||
var ta2 = Run(HistoryAggregateType.TimeAverage2, 0, 20, 0, Outage).Single();
|
||||
((double)ta2.Value!).ShouldBe(20d);
|
||||
ta2.StatusCode.ShouldBe(UncertainCalc);
|
||||
((double)Run(HistoryAggregateType.Total2, 0, 20, 0, Outage).Single().Value!).ShouldBe(200d);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void TimeAverage_and_Total_interpolate_across_the_bad_interval()
|
||||
{
|
||||
// Bridged: [0,15) 10→30, mean 20 ⇒ 300; [15,20) 30 ⇒ 150. TimeAverage = 450/20 = 22.5; Total = 22.5 × 20 = 450.
|
||||
var ta = Run(HistoryAggregateType.TimeAverage, 0, 20, 0, Outage).Single();
|
||||
((double)ta.Value!).ShouldBe(22.5d);
|
||||
ta.StatusCode.ShouldBe(UncertainCalc);
|
||||
((double)Run(HistoryAggregateType.Total, 0, 20, 0, Outage).Single().Value!).ShouldBe(450d);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void An_interval_entirely_inside_an_outage_is_Bad_but_Interpolative_bridges_it()
|
||||
{
|
||||
// Window [10,15): raw is Bad throughout ⇒ data but no non-Bad time ⇒ generic Bad (the SDK rule; BadNoData is
|
||||
// reserved for no data at all). Interpolative at +10 = 10 + 20·(10/15).
|
||||
var ta = Run(HistoryAggregateType.TimeAverage, 10, 15, 0, Outage).Single();
|
||||
ta.StatusCode.ShouldBe(BadCalc);
|
||||
ta.Value.ShouldBeNull();
|
||||
Run(HistoryAggregateType.TimeAverage2, 10, 15, 0, Outage).Single().StatusCode.ShouldBe(BadCalc);
|
||||
Run(HistoryAggregateType.Total2, 10, 15, 0, Outage).Single().StatusCode.ShouldBe(BadCalc);
|
||||
var i = Run(HistoryAggregateType.Interpolative, 10, 15, 0, Outage).Single();
|
||||
((double)i.Value!).ShouldBe(10d + 20d * 10d / 15d, 1e-9);
|
||||
i.StatusCode.ShouldBe(UncertainInterpolated);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Interpolative_on_a_segment_that_runs_into_an_outage_is_uncertain()
|
||||
{
|
||||
// +2 lies between the Good +0 and the Bad +5: bridged 10→30 over [0,15) ⇒ 10 + 20·(2/15); the simple-bounding
|
||||
// signal there is held into an outage, so the point is Uncertain even though both bridging ends are Good.
|
||||
var i = Run(HistoryAggregateType.Interpolative, 2, 20, 0, Outage).Single();
|
||||
((double)i.Value!).ShouldBe(10d + 20d * 2d / 15d, 1e-9);
|
||||
i.StatusCode.ShouldBe(UncertainInterpolated);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void The_latest_usable_bound_is_found_behind_a_bad_bound()
|
||||
{
|
||||
// -10 = 0 G, -5 Bad, +10 = 20 G, +30 = 20 G. Window [0,20) one bucket.
|
||||
// Bridged: -10..10 0→20 ⇒ [0,10) 10→20 (150); [10,20) 20 (200) ⇒ TimeAverage = 17.5.
|
||||
// Raw: [-5,10) Bad; [10,20) 20 Good (200) ⇒ TimeAverage2 = 20. Good 10 s of 20 ⇒ Uncertain.
|
||||
StoredSample[] s =
|
||||
[
|
||||
S(-10, HistorianValue.FromDouble(0), Good), S(-5, HistorianValue.Empty, BadNoCommunication),
|
||||
S(10, HistorianValue.FromDouble(20), Good), S(30, HistorianValue.FromDouble(20), Good),
|
||||
];
|
||||
((double)Run(HistoryAggregateType.TimeAverage, 0, 20, 0, s).Single().Value!).ShouldBe(17.5d);
|
||||
var ta2 = Run(HistoryAggregateType.TimeAverage2, 0, 20, 0, s).Single();
|
||||
((double)ta2.Value!).ShouldBe(20d);
|
||||
ta2.StatusCode.ShouldBe(UncertainCalc);
|
||||
var i = Run(HistoryAggregateType.Interpolative, 0, 20, 0, s).Single();
|
||||
((double)i.Value!).ShouldBe(10d); // inside the outage, bridged -10..10 at 0
|
||||
i.StatusCode.ShouldBe(UncertainInterpolated);
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------- stepped Boolean, Uncertain, invalid inputs
|
||||
|
||||
private static readonly StoredSample[] Switch =
|
||||
[
|
||||
S(0, HistorianValue.FromBoolean(true), Good),
|
||||
S(5, HistorianValue.FromBoolean(false), Good),
|
||||
S(15, HistorianValue.FromBoolean(true), Good),
|
||||
S(25, HistorianValue.FromBoolean(true), Good), // bound after the window
|
||||
];
|
||||
|
||||
[Fact]
|
||||
public void Booleans_are_stepped_not_sloped()
|
||||
{
|
||||
// Window [0,20), 10 s. B0: 1 over [0,5), 0 over [5,10) ⇒ 0.5 (sloped would give 0.375). B1: 0 over [10,15),
|
||||
// 1 over [15,20) ⇒ 0.5. All Good. Total: 5 and 5.
|
||||
var s = Run(HistoryAggregateType.TimeAverage, 0, 20, 10, Switch);
|
||||
Values(s, 0.5, 0.5);
|
||||
s.ShouldAllBe(x => x.StatusCode == GoodCalc);
|
||||
Values(Run(HistoryAggregateType.Total, 0, 20, 10, Switch), 5, 5);
|
||||
// Interpolative at +10: stepped from +5 (false) ⇒ 0, Good.
|
||||
var i = Run(HistoryAggregateType.Interpolative, 0, 20, 10, Switch);
|
||||
Values(i, 1, 0);
|
||||
i.Select(x => x.StatusCode).ShouldBe([GoodRaw, GoodInterpolated]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Uncertain_time_counts_as_good_time()
|
||||
{
|
||||
// TreatUncertainAsBad = false (the SDK's time-based status rule): Uncertain time counts toward PercentDataGood.
|
||||
// 0 = 4 Uncertain, 10 = 6 Good, 20 = 6 Good: [0,10) sloped 4→6 ⇒ mean 5; [10,20) 6. Window [0,20), 10 s.
|
||||
StoredSample[] s =
|
||||
[
|
||||
S(0, HistorianValue.FromDouble(4), Uncertain), S(10, HistorianValue.FromDouble(6), Good),
|
||||
S(20, HistorianValue.FromDouble(6), Good),
|
||||
];
|
||||
var r = Run(HistoryAggregateType.TimeAverage2, 0, 20, 10, s);
|
||||
Values(r, 5, 6);
|
||||
r.Select(x => x.StatusCode).ShouldBe([GoodCalc, GoodCalc]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Int64_samples_are_sloped_like_doubles()
|
||||
{
|
||||
StoredSample[] s = [S(0, HistorianValue.FromInt64(0), Good), S(10, HistorianValue.FromInt64(10), Good)];
|
||||
Values(Run(HistoryAggregateType.TimeAverage2, 0, 10, 0, s), 5);
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData(HistoryAggregateType.TimeAverage)]
|
||||
[InlineData(HistoryAggregateType.TimeAverage2)]
|
||||
[InlineData(HistoryAggregateType.Total)]
|
||||
[InlineData(HistoryAggregateType.Total2)]
|
||||
public void A_usable_string_sample_makes_the_interval_invalid_inputs(HistoryAggregateType agg)
|
||||
{
|
||||
StoredSample[] s = [S(0, HistorianValue.FromString("a"), Good), S(10, HistorianValue.FromString("b"), Good)];
|
||||
var r = Run(agg, 0, 20, 0, s).Single();
|
||||
r.StatusCode.ShouldBe(InvalidInputsCalc);
|
||||
r.Value.ShouldBeNull();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Interpolative_on_a_string_is_invalid_inputs()
|
||||
{
|
||||
StoredSample[] s = [S(0, HistorianValue.FromString("a"), Good), S(10, HistorianValue.FromString("b"), Good)];
|
||||
Run(HistoryAggregateType.Interpolative, 0, 20, 10, s).ShouldAllBe(x => x.StatusCode == InvalidInputs);
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------- plan shape
|
||||
|
||||
[Fact]
|
||||
public void A_reverse_window_is_computed_forward_and_returned_descending()
|
||||
{
|
||||
var s = TimeWeightedAggregator.Evaluate(HistoryAggregateType.TimeAverage2,
|
||||
BucketPlan.Create(At(80), At(0), TimeSpan.FromSeconds(20)), A, Now);
|
||||
s.Select(x => x.SourceTimestampUtc).ShouldBe([At(60), At(40), At(20), At(0)]);
|
||||
Values(s, 50, 50, 45, 17.5);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void A_truncated_last_bucket_uses_its_own_duration_for_Total_and_is_Partial()
|
||||
{
|
||||
// Window [0,50), 20 s: B2 = [40,50) is 10 s of the extrapolated 50 ⇒ TimeAverage 50, Total 500.
|
||||
var s = Run(HistoryAggregateType.Total, 0, 50, 20, A);
|
||||
Values(s, 400, 800, 500);
|
||||
// Only the truncated interval carries the Partial bit.
|
||||
s.Select(x => x.StatusCode).ShouldBe([GoodCalc, UncertainCalc, GoodCalcPartial]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void An_outage_with_no_later_sample_is_Bad_to_the_window_end_not_no_data()
|
||||
{
|
||||
// 0 = 10 G, then an outage marker at +5 and nothing after: [10,20) lies wholly inside the still-open outage.
|
||||
StoredSample[] open = [S(0, HistorianValue.FromDouble(10), Good), S(5, HistorianValue.Empty, BadNoCommunication)];
|
||||
Run(HistoryAggregateType.TimeAverage2, 10, 20, 0, open).Single().StatusCode.ShouldBe(BadCalc);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void A_bad_or_no_data_truncated_interval_carries_no_partial_bit()
|
||||
{
|
||||
// Window [0,15), 10 s: B1 = [10,15) is inside the outage ⇒ Bad without Partial.
|
||||
Run(HistoryAggregateType.TimeAverage2, 0, 15, 10, Outage)[1].StatusCode.ShouldBe(BadCalc);
|
||||
Run(HistoryAggregateType.TimeAverage2, 0, 15, 10)[1].StatusCode.ShouldBe(NoDataCalc);
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------- reverse Interpolative (Part 13: anchored at the later time)
|
||||
|
||||
[Fact]
|
||||
public void A_reverse_Interpolative_read_is_anchored_at_its_later_time()
|
||||
{
|
||||
// Request At(20) → At(0), 20 s: one point, at At(20) (not At(0)). Bounded: segment 10..30 is 20 flat, Good.
|
||||
// A forward read of the same window would give the value at At(0) = 10.
|
||||
var s = TimeWeightedAggregator.Evaluate(HistoryAggregateType.Interpolative,
|
||||
BucketPlan.Create(At(20), At(0), TimeSpan.FromSeconds(20)), Bounded, Now);
|
||||
s.Single().SourceTimestampUtc.ShouldBe(At(20));
|
||||
((double)s.Single().Value!).ShouldBe(20d);
|
||||
s.Single().StatusCode.ShouldBe(GoodInterpolated);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void A_reverse_Interpolative_read_steps_back_from_the_later_time_latest_first()
|
||||
{
|
||||
// Tag A, At(80) → At(0), 20 s: points 80, 60, 40, 20 (latest first). 80 and 60 are past the last sample ⇒ 50
|
||||
// extrapolated, Uncertain; 40 exact Good 50; 20 sits on the Bad sample ⇒ bridged 30, Uncertain.
|
||||
var s = TimeWeightedAggregator.Evaluate(HistoryAggregateType.Interpolative,
|
||||
BucketPlan.Create(At(80), At(0), TimeSpan.FromSeconds(20)), A, Now);
|
||||
s.Select(x => x.SourceTimestampUtc).ShouldBe([At(80), At(60), At(40), At(20)]);
|
||||
Values(s, 50, 50, 50, 30);
|
||||
s.Select(x => x.StatusCode).ShouldBe([UncertainInterpolated, UncertainInterpolated, GoodRaw, UncertainInterpolated]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void A_reverse_Interpolative_read_with_a_partial_interval_ends_inside_the_window()
|
||||
{
|
||||
// At(25) → At(0), 10 s: three intervals ⇒ points 25, 15, 5 (never before the window start).
|
||||
// Bounded: -10..10 slope 1/s from 0 ⇒ +5 = 15; 10..30 flat 20 ⇒ +15 = 20, +25 = 20.
|
||||
var s = TimeWeightedAggregator.Evaluate(HistoryAggregateType.Interpolative,
|
||||
BucketPlan.Create(At(25), At(0), TimeSpan.FromSeconds(10)), Bounded, Now);
|
||||
s.Select(x => x.SourceTimestampUtc).ShouldBe([At(25), At(15), At(5)]);
|
||||
Values(s, 20, 20, 15);
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------- raw-row cap helpers
|
||||
|
||||
[Theory]
|
||||
[InlineData(1_000_000, 1_000_001)]
|
||||
[InlineData(1, 2)]
|
||||
[InlineData(0, int.MaxValue)]
|
||||
[InlineData(-5, int.MaxValue)]
|
||||
[InlineData(int.MaxValue, int.MaxValue)]
|
||||
public void The_window_fetch_limit_is_one_past_the_cap(int cap, int expected) =>
|
||||
TimeWeightedAggregator.WindowFetchLimit(cap).ShouldBe(expected);
|
||||
|
||||
[Fact]
|
||||
public void The_window_row_check_fails_only_past_a_positive_cap()
|
||||
{
|
||||
TimeWeightedAggregator.CheckWindowRows(4, 4, "T");
|
||||
TimeWeightedAggregator.CheckWindowRows(5_000_000, 0, "T");
|
||||
var ex = Should.Throw<HistoryReadLimitExceededException>(() => TimeWeightedAggregator.CheckWindowRows(5, 4, "T"));
|
||||
ex.Limit.ShouldBe(4);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void A_duplicate_bound_is_ignored()
|
||||
{
|
||||
// The same row fetched as both "latest usable" and "latest any" bound.
|
||||
var dup = new[] { Bounded[0], Bounded[0], Bounded[1], Bounded[2], Bounded[2] };
|
||||
Values(Run(HistoryAggregateType.TimeAverage2, 0, 20, 10, dup), 15, 20);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void A_non_time_weighted_aggregate_is_refused()
|
||||
{
|
||||
Should.Throw<HistoryAggregateNotSupportedException>(() =>
|
||||
new TimeWeightedAggregator(HistoryAggregateType.Average, BucketPlan.Create(At(0), At(10), TimeSpan.Zero)));
|
||||
}
|
||||
}
|
||||
+112
@@ -0,0 +1,112 @@
|
||||
using Microsoft.Data.SqlClient;
|
||||
using Shouldly;
|
||||
using Xunit;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
|
||||
using ZB.MOM.WW.OtOpcUa.Driver.Historian.Conformance;
|
||||
using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests;
|
||||
|
||||
/// <summary>
|
||||
/// #668, SQL Server only: the modification-trail row is written by the SAME statement that changes
|
||||
/// <c>hist.Sample</c> (<c>OUTPUT … INTO</c>), so a change and its trail row commit or roll back together. The failure
|
||||
/// is forced INSIDE the audit insert (a unique index on the trail that the second modification of one timestamp
|
||||
/// violates), i.e. after the sample row was already changed within the statement — the strongest place to prove it.
|
||||
/// Each test has a positive control: the same operation succeeds, with its trail row, once the index is dropped.
|
||||
/// </summary>
|
||||
[Collection(SqlHistorianCollection.Name)]
|
||||
public sealed class ModifiedHistoryAtomicityTests(SqlHistorianFixture fx)
|
||||
{
|
||||
private static readonly DateTime T1 = new(2026, 1, 1, 0, 0, 1, DateTimeKind.Utc);
|
||||
private static readonly DateTime T2 = T1.AddSeconds(1);
|
||||
private static readonly HistoryUpdateContext Who = ConformanceData.Editor;
|
||||
|
||||
private static HistorizationValue V(DateTime t, double d) => new(t, HistorianValue.FromDouble(d), 0u);
|
||||
|
||||
private async Task ExecAsync(ReadTestDb db, string sql)
|
||||
{
|
||||
await using var c = await db.Conn.OpenAsync(CancellationToken.None);
|
||||
await using var cmd = db.Conn.CreateCommand(c, sql);
|
||||
await cmd.ExecuteNonQueryAsync();
|
||||
}
|
||||
|
||||
private async Task<long> CountTrailAsync(ReadTestDb db)
|
||||
{
|
||||
await using var c = await db.Conn.OpenAsync(CancellationToken.None);
|
||||
await using var cmd = db.Conn.CreateCommand(c, $"SELECT COUNT(*) FROM {db.Conn.Q("SampleModified")}");
|
||||
return Convert.ToInt64(await cmd.ExecuteScalarAsync());
|
||||
}
|
||||
|
||||
private Task BreakTheTrailAsync(ReadTestDb db) => ExecAsync(db,
|
||||
$"CREATE UNIQUE INDEX UX_Break ON {db.Conn.Q("SampleModified")} (TagId, SourceTimeUtc)");
|
||||
|
||||
private Task RepairTheTrailAsync(ReadTestDb db) => ExecAsync(db,
|
||||
$"DROP INDEX UX_Break ON {db.Conn.Q("SampleModified")}");
|
||||
|
||||
private static async Task<double> ValueAtAsync(ReadTestDb db, string tag, DateTime t)
|
||||
{
|
||||
using var src = db.Source();
|
||||
var r = await src.ReadRawAsync(tag, t, t, 0, CancellationToken.None);
|
||||
return (double)r.Samples.ShouldHaveSingleItem().Value!;
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task A_replace_whose_trail_insert_fails_changes_nothing()
|
||||
{
|
||||
var db = await ReadTestDb.CreateAsync(fx);
|
||||
var u = new SqlServerHistorianDataUpdater(db.Conn, new SqlServerTagCatalog(db.Conn, db.Boot));
|
||||
(await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T1, 1)], Who, CancellationToken.None))
|
||||
.ShouldBe([HistorianStatusCodes.GoodEntryInserted]);
|
||||
(await CountTrailAsync(db)).ShouldBe(1);
|
||||
|
||||
await BreakTheTrailAsync(db);
|
||||
await Should.ThrowAsync<SqlException>(
|
||||
() => u.UpdateDataAsync("t", HistoryUpdateMode.Replace, [V(T1, 99)], Who, CancellationToken.None));
|
||||
|
||||
(await ValueAtAsync(db, "t", T1)).ShouldBe(1d, "the sample change rolled back with its trail row");
|
||||
(await CountTrailAsync(db)).ShouldBe(1, "a failed update leaves no trail row");
|
||||
|
||||
// Positive control: the identical replace succeeds, with its trail row, once the trail accepts it.
|
||||
await RepairTheTrailAsync(db);
|
||||
(await u.UpdateDataAsync("t", HistoryUpdateMode.Replace, [V(T1, 99)], Who, CancellationToken.None))
|
||||
.ShouldBe([HistorianStatusCodes.GoodEntryReplaced]);
|
||||
(await ValueAtAsync(db, "t", T1)).ShouldBe(99d);
|
||||
(await CountTrailAsync(db)).ShouldBe(2);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Deletes_whose_trail_insert_fails_delete_nothing()
|
||||
{
|
||||
var db = await ReadTestDb.CreateAsync(fx);
|
||||
var u = new SqlServerHistorianDataUpdater(db.Conn, new SqlServerTagCatalog(db.Conn, db.Boot));
|
||||
await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T1, 1), V(T2, 2)], Who, CancellationToken.None);
|
||||
|
||||
await BreakTheTrailAsync(db);
|
||||
await Should.ThrowAsync<SqlException>(() => u.DeleteAtTimeAsync("t", [T1], Who, CancellationToken.None));
|
||||
await Should.ThrowAsync<SqlException>(() => u.DeleteRawAsync("t", T1, T2.AddSeconds(1), Who, CancellationToken.None));
|
||||
|
||||
(await ValueAtAsync(db, "t", T1)).ShouldBe(1d);
|
||||
(await ValueAtAsync(db, "t", T2)).ShouldBe(2d);
|
||||
(await CountTrailAsync(db)).ShouldBe(2, "only the two inserts");
|
||||
|
||||
await RepairTheTrailAsync(db);
|
||||
(await u.DeleteAtTimeAsync("t", [T1], Who, CancellationToken.None)).ShouldBe([HistorianStatusCodes.Good]);
|
||||
(await u.DeleteRawAsync("t", T1, T2.AddSeconds(1), Who, CancellationToken.None)).ShouldBe(HistorianStatusCodes.Good);
|
||||
(await CountTrailAsync(db)).ShouldBe(4);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task An_over_long_principal_is_truncated_never_refused()
|
||||
{
|
||||
var db = await ReadTestDb.CreateAsync(fx);
|
||||
var u = new SqlServerHistorianDataUpdater(db.Conn, new SqlServerTagCatalog(db.Conn, db.Boot));
|
||||
var who = new HistoryUpdateContext(new string('p', 500), new string('k', 500));
|
||||
|
||||
(await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T1, 1)], who, CancellationToken.None))
|
||||
.ShouldBe([HistorianStatusCodes.GoodEntryInserted]);
|
||||
|
||||
using var src = db.Source();
|
||||
var trail = await src.ReadModifiedAsync("t", T1.AddHours(-1), T1.AddHours(1), 0, null, CancellationToken.None);
|
||||
trail.Values.ShouldHaveSingleItem().UserName.ShouldBe(new string('p', HistoryUpdateContext.MaxIdentityLength));
|
||||
}
|
||||
}
|
||||
+1
-1
@@ -30,7 +30,7 @@ public sealed class RawReadTests(SqlHistorianFixture fx)
|
||||
var h = src.GetHealthSnapshot();
|
||||
h.TotalFailures.ShouldBe(2);
|
||||
h.ConsecutiveFailures.ShouldBe(2);
|
||||
h.LastError!.ShouldContain("missing version 1");
|
||||
h.LastError!.ShouldContain($"missing version {SqlServerSchemaBootstrapper.RequiredVersion}");
|
||||
h.ProcessConnectionOpen.ShouldBeFalse();
|
||||
|
||||
// Another node creates the schema; the same data source then succeeds (bootstrap failure is not latched).
|
||||
|
||||
+3
-2
@@ -44,8 +44,9 @@ internal sealed class ReadTestDb
|
||||
}
|
||||
|
||||
/// <summary>A fresh data source (fresh catalog cache) over this schema; <paramref name="clock"/> is its "now".</summary>
|
||||
public SqlServerHistorianDataSource Source(TimeProvider? clock = null) =>
|
||||
new(Conn, Boot, new SqlServerTagCatalog(Conn, Boot), timeProvider: clock);
|
||||
public SqlServerHistorianDataSource Source(TimeProvider? clock = null,
|
||||
int maxTimeWeightedRawRows = TimeWeightedAggregator.DefaultMaxWindowRows) =>
|
||||
new(Conn, Boot, new SqlServerTagCatalog(Conn, Boot), timeProvider: clock, maxTimeWeightedRawRows: maxTimeWeightedRawRows);
|
||||
|
||||
/// <summary>Inserts a tag row and the given samples. ServerTimeUtc = source time + 1 s (so tests can tell them apart).</summary>
|
||||
public async Task<int> SeedAsync(string tag, params (DateTime T, HistorianValue V, uint Status)[] samples)
|
||||
|
||||
+234
@@ -0,0 +1,234 @@
|
||||
using Microsoft.Data.SqlClient;
|
||||
using Shouldly;
|
||||
using Xunit;
|
||||
using ZB.MOM.WW.OtOpcUa.Runtime.Historian;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests;
|
||||
|
||||
[Collection(SqlHistorianCollection.Name)]
|
||||
public sealed class RetentionTests(SqlHistorianFixture fx)
|
||||
{
|
||||
private static readonly DateTime Now = new(2026, 10, 11, 12, 0, 0, DateTimeKind.Utc);
|
||||
|
||||
private sealed class FixedTime(DateTime utc) : TimeProvider
|
||||
{
|
||||
public override DateTimeOffset GetUtcNow() => new(utc, TimeSpan.Zero);
|
||||
}
|
||||
|
||||
private (SqlHistorianConnection C, SqlServerSchemaBootstrapper B, SqlServerHistorianOptions O, string S) Make()
|
||||
{
|
||||
var s = SqlHistorianFixture.UniqueSchema();
|
||||
var o = new SqlServerHistorianOptions { ConnectionString = fx.ConnectionString, Schema = s };
|
||||
var c = new SqlHistorianConnection(o);
|
||||
return (c, new SqlServerSchemaBootstrapper(c, o), o, s);
|
||||
}
|
||||
|
||||
private async Task ExecAsync(string sql)
|
||||
{
|
||||
await using var c = new SqlConnection(fx.ConnectionString);
|
||||
await c.OpenAsync();
|
||||
await using var cmd = c.CreateCommand();
|
||||
cmd.CommandText = sql;
|
||||
await cmd.ExecuteNonQueryAsync();
|
||||
}
|
||||
|
||||
private async Task<long> CountAsync(string sql)
|
||||
{
|
||||
await using var c = new SqlConnection(fx.ConnectionString);
|
||||
await c.OpenAsync();
|
||||
await using var cmd = c.CreateCommand();
|
||||
cmd.CommandText = sql;
|
||||
return Convert.ToInt64(await cmd.ExecuteScalarAsync());
|
||||
}
|
||||
|
||||
/// <summary>3 tags x (5 old + 2 new) samples, plus 4 old + 1 new events. Retention 30 days at <see cref="Now"/>.</summary>
|
||||
private async Task SeedAsync(SqlServerSchemaBootstrapper b, string s)
|
||||
{
|
||||
await b.EnsureAsync(CancellationToken.None);
|
||||
var old = Now.AddDays(-40).ToString("yyyy-MM-ddTHH:mm:ss");
|
||||
var fresh = Now.AddDays(-1).ToString("yyyy-MM-ddTHH:mm:ss");
|
||||
var sql = new System.Text.StringBuilder();
|
||||
for (var t = 1; t <= 3; t++)
|
||||
{
|
||||
sql.Append($"INSERT [{s}].Tag (Name, DataType, CreatedUtc, UpdatedUtc) VALUES (N'tag{t}', 1, SYSUTCDATETIME(), SYSUTCDATETIME());");
|
||||
}
|
||||
sql.Append($"DECLARE @i int = 0; WHILE @i < 5 BEGIN INSERT [{s}].Sample (TagId, SourceTimeUtc, ValueKind, NumValue, StatusCode, ServerTimeUtc) SELECT TagId, DATEADD(minute, @i, '{old}'), 1, 1.0, 0, SYSUTCDATETIME() FROM [{s}].Tag; SET @i += 1; END;");
|
||||
sql.Append($"SET @i = 0; WHILE @i < 2 BEGIN INSERT [{s}].Sample (TagId, SourceTimeUtc, ValueKind, NumValue, StatusCode, ServerTimeUtc) SELECT TagId, DATEADD(minute, @i, '{fresh}'), 1, 1.0, 0, SYSUTCDATETIME() FROM [{s}].Tag; SET @i += 1; END;");
|
||||
sql.Append($"SET @i = 0; WHILE @i < 4 BEGIN INSERT [{s}].Event (EventId, SourceName, AlarmId, AlarmName, AlarmTypeName, Severity, EventKind, EventTimeUtc, ReceivedTimeUtc) VALUES (CONCAT('old', @i), 's', 'a', 'n', 't', 1, 'Activated', DATEADD(minute, @i, '{old}'), SYSUTCDATETIME()); SET @i += 1; END;");
|
||||
sql.Append($"INSERT [{s}].Event (EventId, SourceName, AlarmId, AlarmName, AlarmTypeName, Severity, EventKind, EventTimeUtc, ReceivedTimeUtc) VALUES ('new0', 's', 'a', 'n', 't', 1, 'Activated', '{fresh}', SYSUTCDATETIME());");
|
||||
// #668 modification trail: 2 old + 1 fresh row per tag (by SOURCE time — the modification time is "now" on all).
|
||||
sql.Append($"SET @i = 0; WHILE @i < 2 BEGIN INSERT [{s}].SampleModified (TagId, SourceTimeUtc, ModificationTimeUtc, UpdateType, Principal, ValueKind, NumValue, StatusCode, ServerTimeUtc) SELECT TagId, DATEADD(minute, @i, '{old}'), SYSUTCDATETIME(), 2, N'p', 1, 1.0, 0, SYSUTCDATETIME() FROM [{s}].Tag; SET @i += 1; END;");
|
||||
sql.Append($"INSERT [{s}].SampleModified (TagId, SourceTimeUtc, ModificationTimeUtc, UpdateType, Principal) SELECT TagId, '{fresh}', SYSUTCDATETIME(), 1, N'p' FROM [{s}].Tag;");
|
||||
await ExecAsync(sql.ToString());
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData(4000)]
|
||||
[InlineData(1)] // forces the delete loop to iterate once per row
|
||||
public async Task Purge_removes_only_rows_older_than_the_window(int batch)
|
||||
{
|
||||
fx.SkipIfUnavailable();
|
||||
var (c, b, _, s) = Make();
|
||||
await SeedAsync(b, s);
|
||||
var identBefore = await CountAsync($"SELECT IDENT_CURRENT('[{s}].SampleModified')");
|
||||
|
||||
var purger = new SqlServerHistorianRetention(c, b, TimeSpan.FromHours(1), 30, batch, time: new FixedTime(Now));
|
||||
var results = await purger.PurgeAsync(CancellationToken.None);
|
||||
|
||||
// #668: a purge never WRITES to the modification trail. The identity seed proves no row was ever inserted —
|
||||
// a row-count check could not, since the trail purge would delete an old-source-time audit row again.
|
||||
(await CountAsync($"SELECT IDENT_CURRENT('[{s}].SampleModified')")).ShouldBe(identBefore);
|
||||
|
||||
results.ShouldBe([
|
||||
new RetentionPurgeResult("Sample", 15),
|
||||
new RetentionPurgeResult("SampleModified", 6),
|
||||
new RetentionPurgeResult("Event", 4),
|
||||
]);
|
||||
(await CountAsync($"SELECT COUNT(*) FROM [{s}].SampleModified")).ShouldBe(3);
|
||||
(await CountAsync($"SELECT COUNT(*) FROM [{s}].SampleModified WHERE SourceTimeUtc < '{Now.AddDays(-30):yyyy-MM-ddTHH:mm:ss}'")).ShouldBe(0);
|
||||
(await CountAsync($"SELECT COUNT(*) FROM [{s}].Sample")).ShouldBe(6);
|
||||
(await CountAsync($"SELECT COUNT(*) FROM [{s}].Sample WHERE SourceTimeUtc < '{Now.AddDays(-30):yyyy-MM-ddTHH:mm:ss}'")).ShouldBe(0);
|
||||
(await CountAsync($"SELECT COUNT(*) FROM [{s}].Event")).ShouldBe(1);
|
||||
(await CountAsync($"SELECT COUNT(*) FROM [{s}].Tag")).ShouldBe(3);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task A_second_purge_is_a_no_op()
|
||||
{
|
||||
fx.SkipIfUnavailable();
|
||||
var (c, b, _, s) = Make();
|
||||
await SeedAsync(b, s);
|
||||
var purger = new SqlServerHistorianRetention(c, b, TimeSpan.FromHours(1), 30, 4000, time: new FixedTime(Now));
|
||||
await purger.PurgeAsync(CancellationToken.None);
|
||||
var again = await purger.PurgeAsync(CancellationToken.None);
|
||||
again.ShouldBe([
|
||||
new RetentionPurgeResult("Sample", 0),
|
||||
new RetentionPurgeResult("SampleModified", 0),
|
||||
new RetentionPurgeResult("Event", 0),
|
||||
]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task A_purger_skips_while_another_holds_the_applock()
|
||||
{
|
||||
fx.SkipIfUnavailable();
|
||||
var (c, b, _, s) = Make();
|
||||
await SeedAsync(b, s);
|
||||
|
||||
// Hold the same session applock the purger takes, on a separate connection.
|
||||
await using var holder = new SqlConnection(fx.ConnectionString);
|
||||
await holder.OpenAsync();
|
||||
await using (var cmd = holder.CreateCommand())
|
||||
{
|
||||
cmd.CommandText = $"EXEC sp_getapplock @Resource = N'OtOpcUa.Historian.Retention.{s}', @LockMode = 'Exclusive', @LockOwner = 'Session', @LockTimeout = 0;";
|
||||
await cmd.ExecuteNonQueryAsync();
|
||||
}
|
||||
|
||||
var purger = new SqlServerHistorianRetention(c, b, TimeSpan.FromHours(1), 30, 4000, time: new FixedTime(Now));
|
||||
(await purger.PurgeAsync(CancellationToken.None)).ShouldBeEmpty();
|
||||
(await CountAsync($"SELECT COUNT(*) FROM [{s}].Sample")).ShouldBe(21);
|
||||
|
||||
await using (var cmd = holder.CreateCommand())
|
||||
{
|
||||
cmd.CommandText = $"EXEC sp_releaseapplock @Resource = N'OtOpcUa.Historian.Retention.{s}', @LockOwner = 'Session';";
|
||||
await cmd.ExecuteNonQueryAsync();
|
||||
}
|
||||
(await purger.PurgeAsync(CancellationToken.None)).ShouldNotBeEmpty();
|
||||
(await CountAsync($"SELECT COUNT(*) FROM [{s}].Sample")).ShouldBe(6);
|
||||
}
|
||||
|
||||
/// <summary>True when a fresh, non-pooled session can take the retention applock (i.e. nobody holds it).</summary>
|
||||
private async Task<bool> LockIsFreeAsync(string s)
|
||||
{
|
||||
var cs = new SqlConnectionStringBuilder(fx.ConnectionString) { Pooling = false }.ConnectionString;
|
||||
await using var c = new SqlConnection(cs);
|
||||
await c.OpenAsync();
|
||||
await using var cmd = c.CreateCommand();
|
||||
cmd.CommandText = $"DECLARE @r int; EXEC @r = sp_getapplock @Resource = N'OtOpcUa.Historian.Retention.{s}', @LockMode = 'Exclusive', @LockOwner = 'Session', @LockTimeout = 0; SELECT @r;";
|
||||
return Convert.ToInt32(await cmd.ExecuteScalarAsync()) >= 0;
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task The_applock_is_released_after_a_purge()
|
||||
{
|
||||
fx.SkipIfUnavailable();
|
||||
var (c, b, _, s) = Make();
|
||||
await SeedAsync(b, s);
|
||||
var purger = new SqlServerHistorianRetention(c, b, TimeSpan.FromHours(1), 30, 4000, time: new FixedTime(Now));
|
||||
await purger.PurgeAsync(CancellationToken.None);
|
||||
// A pooled-session re-entry would hide a missing release; a separate non-pooled session cannot.
|
||||
(await LockIsFreeAsync(s)).ShouldBeTrue();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Cancelling_between_batches_stops_the_purge_and_still_releases_the_applock()
|
||||
{
|
||||
fx.SkipIfUnavailable();
|
||||
var (c, b, _, s) = Make();
|
||||
await SeedAsync(b, s);
|
||||
using var cts = new CancellationTokenSource();
|
||||
var purger = new SqlServerHistorianRetention(c, b, TimeSpan.FromHours(1), 30, 1, time: new FixedTime(Now));
|
||||
var batches = 0;
|
||||
purger.AfterBatch = () => { if (++batches == 2) cts.Cancel(); };
|
||||
|
||||
await Should.ThrowAsync<OperationCanceledException>(() => purger.PurgeAsync(cts.Token));
|
||||
|
||||
batches.ShouldBe(2);
|
||||
(await CountAsync($"SELECT COUNT(*) FROM [{s}].Sample")).ShouldBe(19); // 21 - two batches of 1
|
||||
(await LockIsFreeAsync(s)).ShouldBeTrue();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task A_throwing_purge_releases_the_applock()
|
||||
{
|
||||
fx.SkipIfUnavailable();
|
||||
var (c, b, _, s) = Make();
|
||||
await SeedAsync(b, s);
|
||||
var purger = new SqlServerHistorianRetention(c, b, TimeSpan.FromHours(1), 30, 1, time: new FixedTime(Now));
|
||||
purger.AfterBatch = () => throw new InvalidOperationException("boom");
|
||||
await Should.ThrowAsync<InvalidOperationException>(() => purger.PurgeAsync(CancellationToken.None));
|
||||
(await LockIsFreeAsync(s)).ShouldBeTrue();
|
||||
}
|
||||
|
||||
private sealed class CapturingLogger : Microsoft.Extensions.Logging.ILogger
|
||||
{
|
||||
public readonly List<Microsoft.Extensions.Logging.LogLevel> Levels = [];
|
||||
public IDisposable? BeginScope<TState>(TState state) where TState : notnull => null;
|
||||
public bool IsEnabled(Microsoft.Extensions.Logging.LogLevel logLevel) => true;
|
||||
public void Log<TState>(Microsoft.Extensions.Logging.LogLevel logLevel, Microsoft.Extensions.Logging.EventId eventId,
|
||||
TState state, Exception? exception, Func<TState, Exception?, string> formatter)
|
||||
{
|
||||
lock (Levels) Levels.Add(logLevel);
|
||||
}
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task A_skipped_pass_logs_no_warning_and_does_not_clear_the_pool()
|
||||
{
|
||||
fx.SkipIfUnavailable();
|
||||
var (c, b, _, s) = Make();
|
||||
await SeedAsync(b, s);
|
||||
|
||||
await using var holder = new SqlConnection(fx.ConnectionString);
|
||||
await holder.OpenAsync();
|
||||
await using (var cmd = holder.CreateCommand())
|
||||
{
|
||||
cmd.CommandText = $"EXEC sp_getapplock @Resource = N'OtOpcUa.Historian.Retention.{s}', @LockMode = 'Exclusive', @LockOwner = 'Session', @LockTimeout = 0;";
|
||||
await cmd.ExecuteNonQueryAsync();
|
||||
}
|
||||
|
||||
var log = new CapturingLogger();
|
||||
var clears = 0;
|
||||
var purger = new SqlServerHistorianRetention(c, b, TimeSpan.FromHours(1), 30, 4000, log, new FixedTime(Now))
|
||||
{
|
||||
ClearPool = _ => Interlocked.Increment(ref clears),
|
||||
};
|
||||
|
||||
for (var i = 0; i < 3; i++) (await purger.PurgeAsync(CancellationToken.None)).ShouldBeEmpty();
|
||||
|
||||
lock (log.Levels) log.Levels.ShouldNotContain(l => l >= Microsoft.Extensions.Logging.LogLevel.Warning);
|
||||
clears.ShouldBe(0);
|
||||
// The shared pool is still healthy.
|
||||
(await CountAsync($"SELECT COUNT(*) FROM [{s}].Sample")).ShouldBe(21);
|
||||
}
|
||||
}
|
||||
+44
-4
@@ -32,7 +32,7 @@ public sealed class SchemaBootstrapIntegrationTests(SqlHistorianFixture fx)
|
||||
var s = SqlHistorianFixture.UniqueSchema();
|
||||
await Boot(Opts(s)).EnsureAsync(CancellationToken.None);
|
||||
await Boot(Opts(s)).EnsureAsync(CancellationToken.None); // fresh instance => re-runs the guarded path
|
||||
(await ScalarAsync($"SELECT COUNT(*) FROM [{s}].SchemaVersion")).ShouldBe(1);
|
||||
(await ScalarAsync($"SELECT COUNT(*) FROM [{s}].SchemaVersion")).ShouldBe(SqlServerSchemaBootstrapper.RequiredVersion);
|
||||
(await ScalarAsync($"SELECT COUNT(*) FROM sys.indexes WHERE name='CX_Event_Time' AND object_id=OBJECT_ID('[{s}].Event')")).ShouldBe(1);
|
||||
}
|
||||
|
||||
@@ -51,7 +51,47 @@ public sealed class SchemaBootstrapIntegrationTests(SqlHistorianFixture fx)
|
||||
cmd.CommandText = b;
|
||||
await cmd.ExecuteNonQueryAsync();
|
||||
}
|
||||
(await ScalarAsync($"SELECT COUNT(*) FROM [{s}].SchemaVersion")).ShouldBe(1);
|
||||
(await ScalarAsync($"SELECT COUNT(*) FROM [{s}].SchemaVersion")).ShouldBe(SqlServerSchemaBootstrapper.RequiredVersion);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// #668 upgrade path: a schema left at version 1 (only <c>001_initial.sql</c> applied, as every pre-#668 install is)
|
||||
/// is upgraded in place by an auto-creating bootstrap — <c>SampleModified</c> appears, both version rows exist, and
|
||||
/// existing samples are untouched. With <c>AutoCreateSchema=false</c> the same v1 schema is refused.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public async Task A_version_1_schema_is_upgraded_to_version_2_in_place()
|
||||
{
|
||||
fx.SkipIfUnavailable();
|
||||
var s = SqlHistorianFixture.UniqueSchema();
|
||||
var v1 = new StreamReader(typeof(SqlServerSchemaBootstrapper).Assembly.GetManifestResourceStream("Schema.001_initial.sql")!).ReadToEnd();
|
||||
await using (var c = new SqlConnection(fx.ConnectionString))
|
||||
{
|
||||
await c.OpenAsync();
|
||||
foreach (var b in SqlServerSchemaBootstrapper.SplitBatches(v1.Replace("$(Schema)", s)))
|
||||
{
|
||||
await using var cmd = c.CreateCommand();
|
||||
cmd.CommandText = b;
|
||||
await cmd.ExecuteNonQueryAsync();
|
||||
}
|
||||
await using var seed = c.CreateCommand();
|
||||
seed.CommandText = $"INSERT [{s}].Tag (Name, DataType, CreatedUtc, UpdatedUtc) VALUES (N'old', 1, SYSUTCDATETIME(), SYSUTCDATETIME()); " +
|
||||
$"INSERT [{s}].Sample (TagId, SourceTimeUtc, ValueKind, NumValue, StatusCode, ServerTimeUtc) SELECT TagId, '2026-01-01', 1, 4.0, 0, SYSUTCDATETIME() FROM [{s}].Tag";
|
||||
await seed.ExecuteNonQueryAsync();
|
||||
}
|
||||
(await ScalarAsync($"SELECT COUNT(*) FROM sys.tables WHERE name = 'SampleModified' AND schema_id = SCHEMA_ID('{s}')")).ShouldBe(0);
|
||||
|
||||
var refused = await Should.ThrowAsync<HistorianSchemaMissingException>(
|
||||
() => Boot(Opts(s, auto: false)).EnsureAsync(CancellationToken.None));
|
||||
refused.Message.ShouldContain("missing version 2 (found 1)");
|
||||
|
||||
await Boot(Opts(s)).EnsureAsync(CancellationToken.None);
|
||||
|
||||
(await ScalarAsync($"SELECT COUNT(*) FROM [{s}].SchemaVersion")).ShouldBe(2);
|
||||
(await ScalarAsync($"SELECT MAX(Version) FROM [{s}].SchemaVersion")).ShouldBe(2);
|
||||
(await ScalarAsync($"SELECT COUNT(*) FROM [{s}].SampleModified")).ShouldBe(0);
|
||||
(await ScalarAsync($"SELECT COUNT(*) FROM [{s}].Sample")).ShouldBe(1);
|
||||
await Boot(Opts(s, auto: false)).EnsureAsync(CancellationToken.None); // now satisfies the verify-only path
|
||||
}
|
||||
|
||||
[Fact]
|
||||
@@ -61,7 +101,7 @@ public sealed class SchemaBootstrapIntegrationTests(SqlHistorianFixture fx)
|
||||
var s = SqlHistorianFixture.UniqueSchema();
|
||||
var tasks = Enumerable.Range(0, 4).Select(_ => Task.Run(() => Boot(Opts(s)).EnsureAsync(CancellationToken.None))).ToArray();
|
||||
await Task.WhenAll(tasks);
|
||||
(await ScalarAsync($"SELECT COUNT(*) FROM [{s}].SchemaVersion")).ShouldBe(1);
|
||||
(await ScalarAsync($"SELECT COUNT(*) FROM [{s}].SchemaVersion")).ShouldBe(SqlServerSchemaBootstrapper.RequiredVersion);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
@@ -70,7 +110,7 @@ public sealed class SchemaBootstrapIntegrationTests(SqlHistorianFixture fx)
|
||||
fx.SkipIfUnavailable();
|
||||
var ex = await Should.ThrowAsync<HistorianSchemaMissingException>(
|
||||
() => Boot(Opts(SqlHistorianFixture.UniqueSchema(), auto: false)).EnsureAsync(CancellationToken.None));
|
||||
ex.Message.ShouldContain("missing version 1");
|
||||
ex.Message.ShouldContain("missing version 2");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
|
||||
+7
@@ -53,3 +53,10 @@ public sealed class SqlServerTagNameConformance(SqlHistorianFixture fx) : TagNam
|
||||
/// <inheritdoc/>
|
||||
protected override Task<IConformanceBackend> CreateBackendAsync() => SqlServerConformanceBackend.CreateAsync(fx);
|
||||
}
|
||||
|
||||
[Collection(SqlHistorianCollection.Name)]
|
||||
public sealed class SqlServerModifiedValueConformance(SqlHistorianFixture fx) : ModifiedValueConformance
|
||||
{
|
||||
/// <inheritdoc/>
|
||||
protected override Task<IConformanceBackend> CreateBackendAsync() => SqlServerConformanceBackend.CreateAsync(fx);
|
||||
}
|
||||
|
||||
+3
@@ -25,6 +25,9 @@ internal sealed class SqlServerConformanceBackend : IConformanceBackend
|
||||
/// <inheritdoc/>
|
||||
public IHistorianDataSource CreateDataSource(TimeProvider? clock = null) => _db.Source(clock);
|
||||
|
||||
/// <inheritdoc/>
|
||||
public IHistorianDataSource CreateDataSource(int maxTimeWeightedRawRows) => _db.Source(null, maxTimeWeightedRawRows);
|
||||
|
||||
/// <inheritdoc/>
|
||||
public IHistorianValueWriter CreateValueWriter(Func<DateTime>? utcNow = null) =>
|
||||
new SqlServerHistorianValueWriter(_db.Conn, _db.Boot, new SqlServerTagCatalog(_db.Conn, _db.Boot), utcNow: utcNow);
|
||||
|
||||
@@ -29,4 +29,30 @@ public sealed class SchemaScriptTests
|
||||
dir.ShouldNotBeNull();
|
||||
File.ReadAllText(Path.Combine(dir!.FullName, "docs/historian/sqlserver/001_initial.sql")).ShouldBe(text);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// #668: schema v2 (the Part 11 modification trail) is embedded, matches its docs copy, records version 2 (which
|
||||
/// must equal <see cref="SqlServerSchemaBootstrapper.RequiredVersion"/>), and carries none of the constructs an
|
||||
/// <c>OUTPUT … INTO</c> target forbids (FOREIGN KEY / CHECK / trigger) — any of them would fail every HistoryUpdate.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void Schema_v2_sample_modified_is_embedded_matches_the_docs_copy_and_is_a_legal_output_into_target()
|
||||
{
|
||||
var asm = typeof(SqlServerSchemaBootstrapper).Assembly;
|
||||
using var s = asm.GetManifestResourceStream("Schema.002_sample_modified.sql");
|
||||
s.ShouldNotBeNull();
|
||||
var text = new StreamReader(s!).ReadToEnd();
|
||||
text.ShouldContain("[SampleModified]");
|
||||
text.ShouldContain("PRIMARY KEY CLUSTERED (TagId, SourceTimeUtc, ModifiedId)");
|
||||
text.ShouldContain("VALUES (2, SYSUTCDATETIME())");
|
||||
SqlServerSchemaBootstrapper.RequiredVersion.ShouldBe(2);
|
||||
var ddl = string.Join('\n', text.Split('\n').Where(l => !l.TrimStart().StartsWith("--", StringComparison.Ordinal)));
|
||||
ddl.ShouldNotContain("FOREIGN KEY", Case.Insensitive);
|
||||
ddl.ShouldNotContain("CHECK", Case.Insensitive);
|
||||
ddl.ShouldNotContain("TRIGGER", Case.Insensitive);
|
||||
|
||||
var dir = new DirectoryInfo(AppContext.BaseDirectory);
|
||||
while (dir is not null && !File.Exists(Path.Combine(dir.FullName, "ZB.MOM.WW.OtOpcUa.slnx"))) dir = dir.Parent;
|
||||
File.ReadAllText(Path.Combine(dir!.FullName, "docs/historian/sqlserver/002_sample_modified.sql")).ShouldBe(text);
|
||||
}
|
||||
}
|
||||
|
||||
+62
@@ -0,0 +1,62 @@
|
||||
using Shouldly;
|
||||
using Xunit;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.Tests;
|
||||
|
||||
public sealed class SqlServerHistorianRetentionOptionsTests
|
||||
{
|
||||
private static SqlServerHistorianOptions Valid() => new() { ConnectionString = "Server=x" };
|
||||
|
||||
[Fact]
|
||||
public void Retention_is_off_by_default_with_sane_defaults()
|
||||
{
|
||||
var o = Valid();
|
||||
o.RetentionDays.ShouldBeNull();
|
||||
o.PurgeInterval.ShouldBe(TimeSpan.FromHours(1));
|
||||
o.PurgeBatchSize.ShouldBe(4000);
|
||||
o.Validate().ShouldBeEmpty();
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData(0)]
|
||||
[InlineData(-5)]
|
||||
public void Non_positive_RetentionDays_is_rejected(int days)
|
||||
{
|
||||
var o = Valid();
|
||||
o.RetentionDays = days;
|
||||
o.Validate().ShouldContain(e => e.Contains("RetentionDays"));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Positive_RetentionDays_is_valid()
|
||||
{
|
||||
var o = Valid();
|
||||
o.RetentionDays = 30;
|
||||
o.Validate().ShouldBeEmpty();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Bad_interval_and_batch_are_rejected()
|
||||
{
|
||||
var o = Valid();
|
||||
o.PurgeInterval = TimeSpan.Zero;
|
||||
o.PurgeBatchSize = 0;
|
||||
var errors = o.Validate();
|
||||
errors.ShouldContain(e => e.Contains("PurgeInterval"));
|
||||
errors.ShouldContain(e => e.Contains("PurgeBatchSize"));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Provider_creates_retention_only_when_RetentionDays_set()
|
||||
{
|
||||
var o = Valid();
|
||||
new SqlServerHistorianProvider(o).CreateRetention(new EmptyServices()).ShouldBeNull();
|
||||
o.RetentionDays = 7;
|
||||
new SqlServerHistorianProvider(o).CreateRetention(new EmptyServices()).ShouldNotBeNull();
|
||||
}
|
||||
|
||||
private sealed class EmptyServices : IServiceProvider
|
||||
{
|
||||
public object? GetService(Type serviceType) => null;
|
||||
}
|
||||
}
|
||||
+3
-7
@@ -14,7 +14,7 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient.Tests;
|
||||
[Trait("Category", "Unit")]
|
||||
public sealed class OpcUaClientDiagnosticsTests
|
||||
{
|
||||
private sealed class Capture : IDisposable
|
||||
internal sealed class Capture : IDisposable
|
||||
{
|
||||
private readonly MeterListener _listener = new();
|
||||
private readonly object _gate = new();
|
||||
@@ -57,9 +57,7 @@ public sealed class OpcUaClientDiagnosticsTests
|
||||
d.Record(OpcUaClientDiagnostics.BadStatusValue);
|
||||
d.Record(OpcUaClientDiagnostics.PublishError);
|
||||
d.Record(OpcUaClientDiagnostics.KeepAliveMiss);
|
||||
d.Record(OpcUaClientDiagnostics.ReconnectAttempt);
|
||||
d.Record(OpcUaClientDiagnostics.ReconnectSuccess);
|
||||
d.Record(OpcUaClientDiagnostics.ReconnectExhausted);
|
||||
d.Record(OpcUaClientDiagnostics.KeepAliveRecovered);
|
||||
|
||||
cap[OpcUaClientDiagnostics.PublishResponse].ShouldBe(1);
|
||||
cap[OpcUaClientDiagnostics.Notification].ShouldBe(3);
|
||||
@@ -67,9 +65,7 @@ public sealed class OpcUaClientDiagnosticsTests
|
||||
cap[OpcUaClientDiagnostics.BadStatusValue].ShouldBe(1);
|
||||
cap[OpcUaClientDiagnostics.PublishError].ShouldBe(1);
|
||||
cap[OpcUaClientDiagnostics.KeepAliveMiss].ShouldBe(1);
|
||||
cap[OpcUaClientDiagnostics.ReconnectAttempt].ShouldBe(1);
|
||||
cap[OpcUaClientDiagnostics.ReconnectSuccess].ShouldBe(1);
|
||||
cap[OpcUaClientDiagnostics.ReconnectExhausted].ShouldBe(1);
|
||||
cap[OpcUaClientDiagnostics.KeepAliveRecovered].ShouldBe(1);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
|
||||
+6
-8
@@ -4,9 +4,10 @@ using Xunit;
|
||||
namespace ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// Scaffold tests for <see cref="SessionReconnectHandler"/> wiring. Wire-level
|
||||
/// disconnect-reconnect-resume coverage against a live upstream server lands with the
|
||||
/// in-process fixture — too much machinery for a unit-test-only lane.
|
||||
/// Scaffold tests for the reconnect options. Since Gitea #670 the driver has no in-driver reconnect
|
||||
/// handler (session loss is reported and the host's comms-loss watchdog re-initialises the driver);
|
||||
/// <c>ReconnectPeriod</c> is retained for config compatibility only. Outage behaviour is covered by
|
||||
/// <see cref="OpcUaClientSessionLossTests"/> and, over the wire, <c>OpcUaClientDriverReconnectWireTests</c>.
|
||||
/// </summary>
|
||||
[Trait("Category", "Unit")]
|
||||
public sealed class OpcUaClientReconnectTests
|
||||
@@ -26,13 +27,10 @@ public sealed class OpcUaClientReconnectTests
|
||||
opts.ReconnectPeriod.ShouldBe(TimeSpan.FromMilliseconds(500));
|
||||
}
|
||||
|
||||
/// <summary>Verifies that the driver starts with no reconnect handler active before initialization.</summary>
|
||||
/// <summary>Verifies that the driver starts in the Unknown state before initialization.</summary>
|
||||
[Fact]
|
||||
public void Driver_starts_with_no_reconnect_handler_active_pre_init()
|
||||
public void Driver_starts_unknown_pre_init()
|
||||
{
|
||||
// The reconnect handler is lazy — spun up only when a bad keep-alive fires. Pre-init
|
||||
// there's no session to reconnect, so the field must be null (indirectly verified by
|
||||
// the lifecycle-shape test suite catching any accidental construction).
|
||||
using var drv = new OpcUaClientDriver(new OpcUaClientDriverOptions(), "opcua-reconnect");
|
||||
drv.GetHealth().State.ShouldBe(Core.Abstractions.DriverState.Unknown);
|
||||
}
|
||||
|
||||
+722
@@ -0,0 +1,722 @@
|
||||
using System.Text.Json;
|
||||
using Moq;
|
||||
using Opc.Ua;
|
||||
using Opc.Ua.Client;
|
||||
using Shouldly;
|
||||
using Xunit;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// Gitea #670 — upstream session loss must reach the driver's health and its subscribers. Before the fix a bad
|
||||
/// keep-alive only flipped the host status: the subscribed values stayed Good forever, health stayed Healthy (so the
|
||||
/// #527 comms-loss watchdog could never trip — its first conjunct is "non-Healthy"), and <c>LastSuccessfulRead</c>
|
||||
/// only moved on init / <c>ReadAsync</c>, which a subscription-only driver never calls.
|
||||
/// </summary>
|
||||
[Trait("Category", "Unit")]
|
||||
public sealed class OpcUaClientSessionLossTests
|
||||
{
|
||||
private const uint BadNoCommunication = 0x80310000u;
|
||||
private const string SpeedRef = "Plant/dev/Speed";
|
||||
private const string CounterRef = "Plant/dev/Counter";
|
||||
|
||||
private sealed class ManualClock(DateTimeOffset start) : TimeProvider
|
||||
{
|
||||
public DateTimeOffset Now { get; set; } = start;
|
||||
public override DateTimeOffset GetUtcNow() => Now;
|
||||
public void Advance(TimeSpan by) => Now += by;
|
||||
}
|
||||
|
||||
private static RawTagEntry Tag(string rawPath, string nodeId) =>
|
||||
new(rawPath, JsonSerializer.Serialize(new { nodeId }), WriteIdempotent: true);
|
||||
|
||||
private static readonly IReadOnlyList<RawTagEntry> Tags =
|
||||
[
|
||||
Tag(SpeedRef, "ns=0;s=Speed"),
|
||||
Tag(CounterRef, "ns=0;s=Counter"),
|
||||
];
|
||||
|
||||
private static Mock<ISession> NewSession()
|
||||
{
|
||||
var mock = new Mock<ISession>(MockBehavior.Loose);
|
||||
mock.SetupGet(s => s.MessageContext)
|
||||
.Returns(new ServiceMessageContext(SdkTelemetry.Context));
|
||||
mock.SetupGet(s => s.NamespaceUris).Returns(new NamespaceTable());
|
||||
return mock;
|
||||
}
|
||||
|
||||
private static (OpcUaClientDriver Driver, Mock<ISession> Session, ManualClock Clock) NewConnectedDriver(
|
||||
string driverInstanceId = "opcua-670")
|
||||
{
|
||||
var drv = new OpcUaClientDriver(new OpcUaClientDriverOptions { RawTags = Tags }, driverInstanceId);
|
||||
var session = NewSession();
|
||||
var clock = new ManualClock(new DateTimeOffset(2026, 10, 11, 12, 0, 0, TimeSpan.Zero));
|
||||
drv.Clock = clock;
|
||||
drv.SetSessionForTest(session.Object);
|
||||
drv.SetNamespaceMapForTest(NamespaceMap.FromTable(new NamespaceTable(), Tags));
|
||||
drv.SetHealthForTest(new DriverHealth(DriverState.Healthy, clock.Now.UtcDateTime, null));
|
||||
return (drv, session, clock);
|
||||
}
|
||||
|
||||
private static DataValue Good(object value, DateTime ts) =>
|
||||
new(new Variant(value), StatusCodes.Good, ts, ts);
|
||||
|
||||
private static void SetupRead(Mock<ISession> session, params object[] values) =>
|
||||
session.Setup(s => s.ReadAsync(
|
||||
It.IsAny<RequestHeader>(), It.IsAny<double>(), It.IsAny<TimestampsToReturn>(),
|
||||
It.IsAny<ReadValueIdCollection>(), It.IsAny<CancellationToken>()))
|
||||
.ReturnsAsync((RequestHeader _, double _, TimestampsToReturn _, ReadValueIdCollection nodes, CancellationToken _) =>
|
||||
new ReadResponse
|
||||
{
|
||||
ResponseHeader = new ResponseHeader(),
|
||||
Results = [.. nodes.Select((_, i) => new DataValue(new Variant(values[i]), StatusCodes.Good, DateTime.UtcNow, DateTime.UtcNow))],
|
||||
DiagnosticInfos = [],
|
||||
});
|
||||
|
||||
private static async Task WaitForAsync(Func<bool> condition, string because)
|
||||
{
|
||||
var deadline = DateTime.UtcNow + TimeSpan.FromSeconds(10);
|
||||
while (DateTime.UtcNow < deadline)
|
||||
{
|
||||
if (condition()) return;
|
||||
await Task.Delay(10);
|
||||
}
|
||||
condition().ShouldBeTrue(because);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void A_bad_keep_alive_degrades_health_and_preserves_the_last_successful_read()
|
||||
{
|
||||
var (drv, _, clock) = NewConnectedDriver();
|
||||
using var _ = drv;
|
||||
var before = drv.GetHealth().LastSuccessfulRead;
|
||||
|
||||
clock.Advance(TimeSpan.FromSeconds(5));
|
||||
drv.RaiseKeepAliveForTest(StatusCodes.BadNoCommunication);
|
||||
|
||||
var h = drv.GetHealth();
|
||||
h.State.ShouldNotBe(DriverState.Healthy, "a lost session must be visible to the #527 watchdog's first conjunct");
|
||||
h.State.ShouldBe(DriverState.Degraded);
|
||||
h.LastSuccessfulRead.ShouldBe(before, "the watchdog measures silence from this stamp, so loss must not move it");
|
||||
h.LastError.ShouldNotBeNull().ShouldContain("keep-alive");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void A_bad_keep_alive_publishes_BadNoCommunication_once_per_ref_retaining_the_value()
|
||||
{
|
||||
var (drv, _, clock) = NewConnectedDriver();
|
||||
using var _ = drv;
|
||||
var handle = drv.AddSubscriptionForTest([SpeedRef, CounterRef]);
|
||||
var t0 = clock.Now.UtcDateTime;
|
||||
drv.DeliverNotificationForTest(handle, SpeedRef, Good(42.5f, t0));
|
||||
drv.DeliverNotificationForTest(handle, CounterRef, Good(7, t0));
|
||||
|
||||
var seen = new List<DataChangeEventArgs>();
|
||||
drv.OnDataChange += (_, e) => seen.Add(e);
|
||||
|
||||
drv.RaiseKeepAliveForTest(StatusCodes.BadNoCommunication);
|
||||
|
||||
seen.Count.ShouldBe(2, "exactly one Bad per subscribed ref");
|
||||
var speed = seen.Single(e => e.FullReference == SpeedRef);
|
||||
speed.Snapshot.StatusCode.ShouldBe(BadNoCommunication);
|
||||
speed.Snapshot.Value.ShouldBe(42.5f, "the value is retained — only the quality changes (as the host does)");
|
||||
speed.SubscriptionHandle.ShouldBe(handle);
|
||||
seen.Single(e => e.FullReference == CounterRef).Snapshot.Value.ShouldBe(7);
|
||||
|
||||
// The SDK re-raises a bad keep-alive every interval while the channel stays down: no further publishes.
|
||||
drv.RaiseKeepAliveForTest(StatusCodes.BadNoCommunication);
|
||||
drv.RaiseKeepAliveForTest(StatusCodes.BadConnectionClosed);
|
||||
seen.Count.ShouldBe(2, "a second bad keep-alive in the same outage must publish nothing");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Good_keep_alives_and_notifications_advance_the_last_successful_read()
|
||||
{
|
||||
var (drv, _, clock) = NewConnectedDriver();
|
||||
using var _ = drv;
|
||||
var handle = drv.AddSubscriptionForTest([SpeedRef]);
|
||||
|
||||
clock.Advance(TimeSpan.FromSeconds(30));
|
||||
drv.RaiseKeepAliveForTest(StatusCodes.Good);
|
||||
drv.GetHealth().LastSuccessfulRead.ShouldBe(clock.Now.UtcDateTime);
|
||||
|
||||
clock.Advance(TimeSpan.FromSeconds(30));
|
||||
drv.DeliverNotificationForTest(handle, SpeedRef, Good(1.0f, clock.Now.UtcDateTime));
|
||||
drv.GetHealth().LastSuccessfulRead.ShouldBe(clock.Now.UtcDateTime);
|
||||
drv.GetHealth().State.ShouldBe(DriverState.Healthy);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Falsification control for the R2 half of the fix: a long-running, idle, subscription-only driver whose
|
||||
/// server is perfectly alive must never look dead to the #527 watchdog (non-Healthy AND silent ≥ 90 s).
|
||||
/// Without good keep-alives advancing <c>LastSuccessfulRead</c>, fixing R1 alone would trip it on the first
|
||||
/// bad tick of any connection older than 90 s.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void An_idle_healthy_subscription_driver_never_looks_dead_to_the_watchdog()
|
||||
{
|
||||
var (drv, _, clock) = NewConnectedDriver();
|
||||
using var _ = drv;
|
||||
drv.AddSubscriptionForTest([SpeedRef]); // static value — no notifications ever arrive
|
||||
|
||||
for (var i = 0; i < 60; i++) // 5 minutes of 5 s keep-alives
|
||||
{
|
||||
clock.Advance(TimeSpan.FromSeconds(5));
|
||||
drv.RaiseKeepAliveForTest(StatusCodes.Good);
|
||||
var h = drv.GetHealth();
|
||||
h.State.ShouldBe(DriverState.Healthy);
|
||||
(clock.Now.UtcDateTime - h.LastSuccessfulRead!.Value).ShouldBeLessThan(TimeSpan.FromSeconds(90));
|
||||
}
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Recovery_on_the_same_session_republishes_the_identical_value_Good()
|
||||
{
|
||||
var (drv, session, clock) = NewConnectedDriver();
|
||||
using var _ = drv;
|
||||
var handle = drv.AddSubscriptionForTest([SpeedRef]);
|
||||
drv.DeliverNotificationForTest(handle, SpeedRef, Good(42.5f, clock.Now.UtcDateTime));
|
||||
SetupRead(session, 42.5f); // the server still holds the SAME value — it will not resend it
|
||||
|
||||
var seen = new List<DataChangeEventArgs>();
|
||||
drv.OnDataChange += (_, e) => { lock (seen) seen.Add(e); };
|
||||
|
||||
drv.RaiseKeepAliveForTest(StatusCodes.BadNoCommunication);
|
||||
clock.Advance(TimeSpan.FromSeconds(20));
|
||||
drv.RaiseKeepAliveForTest(StatusCodes.Good);
|
||||
|
||||
drv.GetHealth().State.ShouldBe(DriverState.Healthy);
|
||||
drv.GetHealth().LastSuccessfulRead.ShouldBe(clock.Now.UtcDateTime);
|
||||
await WaitForAsync(() => { lock (seen) return seen.Count >= 2; }, "the recovery read must publish");
|
||||
|
||||
lock (seen)
|
||||
{
|
||||
seen.Count.ShouldBe(2);
|
||||
seen[0].Snapshot.StatusCode.ShouldBe(BadNoCommunication);
|
||||
seen[1].FullReference.ShouldBe(SpeedRef);
|
||||
seen[1].SubscriptionHandle.ShouldBe(handle);
|
||||
seen[1].Snapshot.StatusCode.ShouldBe(0u);
|
||||
seen[1].Snapshot.Value.ShouldBe(42.5f, "the IDENTICAL value must come back Good — a (value,status) dedup lets it through");
|
||||
}
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task A_notification_during_recovery_wins_over_the_recovery_read()
|
||||
{
|
||||
var (drv, session, clock) = NewConnectedDriver();
|
||||
using var _ = drv;
|
||||
var handle = drv.AddSubscriptionForTest([SpeedRef]);
|
||||
drv.DeliverNotificationForTest(handle, SpeedRef, Good(1.0f, clock.Now.UtcDateTime));
|
||||
drv.RaiseKeepAliveForTest(StatusCodes.BadNoCommunication);
|
||||
|
||||
// A fresh notification arrives before the keep-alive recovers: the read must not overwrite it.
|
||||
drv.DeliverNotificationForTest(handle, SpeedRef, Good(2.0f, clock.Now.UtcDateTime));
|
||||
SetupRead(session, 1.0f);
|
||||
var seen = new List<DataChangeEventArgs>();
|
||||
drv.OnDataChange += (_, e) => { lock (seen) seen.Add(e); };
|
||||
drv.RaiseKeepAliveForTest(StatusCodes.Good);
|
||||
|
||||
await Task.Delay(200, TestContext.Current.CancellationToken);
|
||||
lock (seen) seen.ShouldBeEmpty();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void A_bad_keep_alive_annotates_subscribed_alarm_conditions_quality_only_once()
|
||||
{
|
||||
var (drv, _, _) = NewConnectedDriver();
|
||||
using var _ = drv;
|
||||
drv.AddAlarmSubscriptionForTest([SpeedRef]);
|
||||
var seen = new List<AlarmEventArgs>();
|
||||
drv.OnAlarmEvent += (_, e) => seen.Add(e);
|
||||
|
||||
drv.RaiseKeepAliveForTest(StatusCodes.BadNoCommunication);
|
||||
drv.RaiseKeepAliveForTest(StatusCodes.BadNoCommunication);
|
||||
|
||||
var e = seen.ShouldHaveSingleItem();
|
||||
e.ConditionId.ShouldBe(SpeedRef);
|
||||
e.IsQualityOnly.ShouldBeTrue("an outage says nothing about the alarm state — quality only, no transition");
|
||||
e.QualityStatusCode.ShouldBe(BadNoCommunication);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Recovery_restores_alarm_condition_quality_Good()
|
||||
{
|
||||
var (drv, session, _) = NewConnectedDriver();
|
||||
using var _ = drv;
|
||||
SetupRead(session);
|
||||
drv.AddAlarmSubscriptionForTest([SpeedRef]);
|
||||
var seen = new List<AlarmEventArgs>();
|
||||
drv.OnAlarmEvent += (_, e) => { lock (seen) seen.Add(e); };
|
||||
|
||||
drv.RaiseKeepAliveForTest(StatusCodes.BadNoCommunication);
|
||||
drv.RaiseKeepAliveForTest(StatusCodes.Good);
|
||||
|
||||
await WaitForAsync(() => { lock (seen) return seen.Count >= 2; }, "recovery must re-annotate Good");
|
||||
lock (seen)
|
||||
{
|
||||
seen[1].IsQualityOnly.ShouldBeTrue();
|
||||
seen[1].QualityStatusCode.ShouldBe(0u);
|
||||
}
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void A_keep_alive_from_a_superseded_session_is_ignored()
|
||||
{
|
||||
var (drv, _, _) = NewConnectedDriver();
|
||||
using var _ = drv;
|
||||
var stale = NewSession();
|
||||
|
||||
drv.RaiseKeepAliveForTest(StatusCodes.BadNoCommunication, stale.Object);
|
||||
|
||||
drv.GetHealth().State.ShouldBe(DriverState.Healthy);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Shutdown_clears_the_outage_so_a_fresh_session_reports_its_own_loss()
|
||||
{
|
||||
var (drv, _, _) = NewConnectedDriver();
|
||||
using var _ = drv;
|
||||
drv.AddSubscriptionForTest([SpeedRef]);
|
||||
drv.RaiseKeepAliveForTest(StatusCodes.BadNoCommunication);
|
||||
|
||||
await drv.ShutdownAsync(TestContext.Current.CancellationToken);
|
||||
|
||||
// As after a watchdog reinit: a new session + subscription, and a new outage must be reported again.
|
||||
var fresh = NewSession();
|
||||
drv.SetSessionForTest(fresh.Object);
|
||||
drv.SetNamespaceMapForTest(NamespaceMap.FromTable(new NamespaceTable(), Tags));
|
||||
drv.AddSubscriptionForTest([SpeedRef]);
|
||||
var seen = new List<DataChangeEventArgs>();
|
||||
drv.OnDataChange += (_, e) => seen.Add(e);
|
||||
drv.RaiseKeepAliveForTest(StatusCodes.BadNoCommunication);
|
||||
|
||||
seen.ShouldHaveSingleItem().Snapshot.StatusCode.ShouldBe(BadNoCommunication);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Review M2: a recovery overtaken by a re-initialise (watchdog Shutdown + Initialize) must not leak its
|
||||
/// outcome onto the NEW session — previously its failed read re-armed the outage latch, so the new session's
|
||||
/// first real outage was swallowed.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public async Task A_recovery_overtaken_by_a_reinit_does_not_latch_the_new_session()
|
||||
{
|
||||
var ct = TestContext.Current.CancellationToken;
|
||||
var (drv, _, clock) = NewConnectedDriver();
|
||||
using var _ = drv;
|
||||
var handle = drv.AddSubscriptionForTest([SpeedRef]);
|
||||
drv.DeliverNotificationForTest(handle, SpeedRef, Good(1.0f, clock.Now.UtcDateTime));
|
||||
drv.RaiseKeepAliveForTest(StatusCodes.BadNoCommunication);
|
||||
|
||||
await drv.Gate.WaitAsync(ct); // park the recovery read on the gate
|
||||
drv.RaiseKeepAliveForTest(StatusCodes.Good);
|
||||
await drv.ShutdownAsync(ct); // the watchdog's re-initialise overtakes it …
|
||||
var fresh = NewSession(); // … and a new session is up (its reads would fail)
|
||||
drv.SetSessionForTest(fresh.Object);
|
||||
drv.SetNamespaceMapForTest(NamespaceMap.FromTable(new NamespaceTable(), Tags));
|
||||
drv.SetHealthForTest(new DriverHealth(DriverState.Healthy, clock.Now.UtcDateTime, null));
|
||||
drv.AddSubscriptionForTest([SpeedRef]);
|
||||
drv.Gate.Release();
|
||||
await drv.RecoveryTaskForTest;
|
||||
|
||||
drv.GetHealth().State.ShouldBe(DriverState.Healthy, "the stale recovery must not degrade the new session");
|
||||
var seen = new List<DataChangeEventArgs>();
|
||||
drv.OnDataChange += (_, e) => seen.Add(e);
|
||||
drv.RaiseKeepAliveForTest(StatusCodes.BadNoCommunication);
|
||||
seen.ShouldHaveSingleItem("the new session's first outage must be published, not swallowed by a stale latch")
|
||||
.Snapshot.StatusCode.ShouldBe(BadNoCommunication);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Review M1: the subscription dies while the session lives (an outage longer than the subscription lifetime
|
||||
/// but shorter than the session timeout). Keep-alives stay Good, so without detection the driver ends Healthy
|
||||
/// with fresh stamps and frozen values forever. It must go Degraded, publish the loss once, and STOP advancing
|
||||
/// the read stamp — so the watchdog's reinit recreates the subscription.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void A_subscription_that_stops_publishing_while_the_session_lives_freezes_the_stamp_and_goes_Bad()
|
||||
{
|
||||
var (drv, _, clock) = NewConnectedDriver();
|
||||
using var _ = drv;
|
||||
var handle = drv.AddSubscriptionForTest([SpeedRef]);
|
||||
var otherHandle = drv.AddSubscriptionForTest([CounterRef]);
|
||||
drv.DeliverNotificationForTest(handle, SpeedRef, Good(3.0f, clock.Now.UtcDateTime));
|
||||
var stalled = false;
|
||||
drv.IsPublishingStopped = sub => stalled && sub.DisplayName == "opcua-sub-1";
|
||||
drv.PublishingStallGrace = _ => TimeSpan.FromSeconds(10);
|
||||
var seen = new List<DataChangeEventArgs>();
|
||||
drv.OnDataChange += (_, e) => seen.Add(e);
|
||||
|
||||
stalled = true;
|
||||
clock.Advance(TimeSpan.FromSeconds(5));
|
||||
drv.RaiseKeepAliveForTest(StatusCodes.Good); // first observation — within grace
|
||||
drv.GetHealth().State.ShouldBe(DriverState.Healthy);
|
||||
clock.Advance(TimeSpan.FromSeconds(10));
|
||||
drv.RaiseKeepAliveForTest(StatusCodes.Good); // grace elapsed
|
||||
|
||||
var h = drv.GetHealth();
|
||||
h.State.ShouldBe(DriverState.Degraded);
|
||||
h.LastError.ShouldNotBeNull().ShouldContain("subscription lost");
|
||||
var frozen = h.LastSuccessfulRead;
|
||||
seen.Count(e => e.Snapshot.StatusCode == BadNoCommunication).ShouldBe(2, "every subscribed ref published Bad once");
|
||||
seen.Single(e => e.FullReference == SpeedRef).Snapshot.Value.ShouldBe(3.0f);
|
||||
|
||||
// Nothing may revive the stamp: not good keep-alives, not a bad→good keep-alive cycle, not notifications
|
||||
// from a still-healthy sibling subscription.
|
||||
for (var i = 0; i < 30; i++)
|
||||
{
|
||||
clock.Advance(TimeSpan.FromSeconds(5));
|
||||
drv.RaiseKeepAliveForTest(StatusCodes.Good);
|
||||
drv.DeliverNotificationForTest(otherHandle, CounterRef, Good(i, clock.Now.UtcDateTime));
|
||||
}
|
||||
drv.RaiseKeepAliveForTest(StatusCodes.BadNoCommunication);
|
||||
drv.RaiseKeepAliveForTest(StatusCodes.Good);
|
||||
|
||||
h = drv.GetHealth();
|
||||
h.State.ShouldBe(DriverState.Degraded);
|
||||
h.LastSuccessfulRead.ShouldBe(frozen, "the #527 watchdog needs a frozen stamp to re-initialise the driver");
|
||||
seen.Count(e => e.Snapshot.StatusCode == BadNoCommunication).ShouldBe(2, "the loss is published once per connection");
|
||||
}
|
||||
|
||||
/// <summary>Control for M1: a subscription that is briefly silent (e.g. right after an outage) and then resumes
|
||||
/// within its grace is NOT declared dead.</summary>
|
||||
[Fact]
|
||||
public void A_subscription_silent_for_less_than_its_grace_is_not_declared_lost()
|
||||
{
|
||||
var (drv, _, clock) = NewConnectedDriver();
|
||||
using var _ = drv;
|
||||
drv.AddSubscriptionForTest([SpeedRef]);
|
||||
var stalled = true;
|
||||
drv.IsPublishingStopped = _ => stalled;
|
||||
drv.PublishingStallGrace = _ => TimeSpan.FromSeconds(10);
|
||||
|
||||
drv.RaiseKeepAliveForTest(StatusCodes.Good);
|
||||
clock.Advance(TimeSpan.FromSeconds(9));
|
||||
drv.RaiseKeepAliveForTest(StatusCodes.Good);
|
||||
stalled = false;
|
||||
for (var i = 0; i < 10; i++)
|
||||
{
|
||||
clock.Advance(TimeSpan.FromSeconds(5));
|
||||
drv.RaiseKeepAliveForTest(StatusCodes.Good);
|
||||
}
|
||||
stalled = true; // a NEW stall restarts the grace rather than inheriting the old one
|
||||
clock.Advance(TimeSpan.FromSeconds(5));
|
||||
drv.RaiseKeepAliveForTest(StatusCodes.Good);
|
||||
|
||||
drv.GetHealth().State.ShouldBe(DriverState.Healthy);
|
||||
drv.GetHealth().LastSuccessfulRead.ShouldBe(clock.Now.UtcDateTime);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void A_BadNoSubscription_publish_error_with_tracked_subscriptions_declares_the_subscription_lost()
|
||||
{
|
||||
var (drv, _, _) = NewConnectedDriver();
|
||||
using var _ = drv;
|
||||
drv.AddSubscriptionForTest([SpeedRef]);
|
||||
var seen = new List<DataChangeEventArgs>();
|
||||
drv.OnDataChange += (_, e) => seen.Add(e);
|
||||
|
||||
drv.RaisePublishErrorForTest(StatusCodes.BadSubscriptionIdInvalid, subscriptionId: 4242); // not ours
|
||||
drv.GetHealth().State.ShouldBe(DriverState.Healthy);
|
||||
|
||||
drv.RaisePublishErrorForTest(StatusCodes.BadNoSubscription);
|
||||
drv.GetHealth().State.ShouldBe(DriverState.Degraded);
|
||||
seen.ShouldHaveSingleItem().Snapshot.StatusCode.ShouldBe(BadNoCommunication);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void A_BadNoSubscription_with_nothing_tracked_is_not_a_loss()
|
||||
{
|
||||
var (drv, _, _) = NewConnectedDriver();
|
||||
using var _ = drv;
|
||||
drv.RaisePublishErrorForTest(StatusCodes.BadNoSubscription);
|
||||
drv.GetHealth().State.ShouldBe(DriverState.Healthy);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Shutdown_clears_a_subscription_loss()
|
||||
{
|
||||
var (drv, _, clock) = NewConnectedDriver();
|
||||
using var _ = drv;
|
||||
drv.AddSubscriptionForTest([SpeedRef]);
|
||||
drv.RaisePublishErrorForTest(StatusCodes.BadNoSubscription);
|
||||
|
||||
await drv.ShutdownAsync(TestContext.Current.CancellationToken);
|
||||
drv.SetSessionForTest(NewSession().Object);
|
||||
drv.SetHealthForTest(new DriverHealth(DriverState.Healthy, clock.Now.UtcDateTime, null));
|
||||
clock.Advance(TimeSpan.FromSeconds(5));
|
||||
drv.RaiseKeepAliveForTest(StatusCodes.Good);
|
||||
|
||||
drv.GetHealth().LastSuccessfulRead.ShouldBe(clock.Now.UtcDateTime, "the re-subscribed connection starts clean");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Sdk_log_forwarding_is_rate_limited()
|
||||
{
|
||||
var target = new CountingLogger();
|
||||
var limited = new SdkTelemetry.RateLimitedLogger(target, max: 3, window: TimeSpan.FromHours(1));
|
||||
for (var i = 0; i < 50; i++)
|
||||
Microsoft.Extensions.Logging.LoggerExtensions.LogWarning(limited, "sdk warning {I}", i);
|
||||
target.Count.ShouldBe(3);
|
||||
}
|
||||
|
||||
private sealed class CountingLogger : Microsoft.Extensions.Logging.ILogger
|
||||
{
|
||||
public int Count;
|
||||
public IDisposable? BeginScope<TState>(TState state) where TState : notnull => null;
|
||||
public bool IsEnabled(Microsoft.Extensions.Logging.LogLevel logLevel) => true;
|
||||
public void Log<TState>(Microsoft.Extensions.Logging.LogLevel logLevel, Microsoft.Extensions.Logging.EventId eventId,
|
||||
TState state, Exception? exception, Func<TState, Exception?, string> formatter) => Count++;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// #670 follow-up F1 (live gate): the outage Bad carried the last Good sample's SOURCE timestamp, so the
|
||||
/// recorder (which historizes <c>SourceTimestampUtc ?? ServerTimestampUtc</c>) wrote it at the Good row's key —
|
||||
/// the historian's insert-if-absent (TagId, SourceTimeUtc) key dropped it and no outage marker was ever stored.
|
||||
/// The Bad is a fresh observation: it is stamped with the detection time, like the host's own comms-loss fan.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void The_outage_Bad_is_stamped_with_the_detection_time_not_the_last_good_source_time()
|
||||
{
|
||||
var (drv, _, clock) = NewConnectedDriver();
|
||||
using var _ = drv;
|
||||
var handle = drv.AddSubscriptionForTest([SpeedRef]);
|
||||
var goodAt = clock.Now.UtcDateTime;
|
||||
drv.DeliverNotificationForTest(handle, SpeedRef, Good(42.5f, goodAt));
|
||||
var seen = new List<DataChangeEventArgs>();
|
||||
drv.OnDataChange += (_, e) => seen.Add(e);
|
||||
|
||||
clock.Advance(TimeSpan.FromSeconds(37));
|
||||
drv.RaiseKeepAliveForTest(StatusCodes.BadNoCommunication);
|
||||
|
||||
var bad = seen.ShouldHaveSingleItem().Snapshot;
|
||||
bad.StatusCode.ShouldBe(BadNoCommunication);
|
||||
bad.Value.ShouldBe(42.5f, "the value is still retained");
|
||||
bad.SourceTimestampUtc.ShouldBe(clock.Now.UtcDateTime, "a Bad at the Good row's source time collides with it in history");
|
||||
bad.ServerTimestampUtc.ShouldBe(clock.Now.UtcDateTime);
|
||||
}
|
||||
|
||||
private static void SetupReadAt(Mock<ISession> session, object value, DateTime sourceTime) =>
|
||||
session.Setup(s => s.ReadAsync(
|
||||
It.IsAny<RequestHeader>(), It.IsAny<double>(), It.IsAny<TimestampsToReturn>(),
|
||||
It.IsAny<ReadValueIdCollection>(), It.IsAny<CancellationToken>()))
|
||||
.ReturnsAsync(new ReadResponse
|
||||
{
|
||||
ResponseHeader = new ResponseHeader(),
|
||||
Results = [new DataValue(new Variant(value), StatusCodes.Good, sourceTime, sourceTime)],
|
||||
DiagnosticInfos = [],
|
||||
});
|
||||
|
||||
/// <summary>Drives a same-session outage + recovery (bad → good keep-alive) and waits for the recovery read.</summary>
|
||||
private static async Task RecoverInPlaceAsync(OpcUaClientDriver drv)
|
||||
{
|
||||
drv.RaiseKeepAliveForTest(StatusCodes.BadNoCommunication);
|
||||
drv.RaiseKeepAliveForTest(StatusCodes.Good);
|
||||
await drv.RecoveryTaskForTest;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// #670 follow-up F2 (live gate): after a same-session recovery the SDK delivered one notification late with
|
||||
/// an OLDER source time, and the node regressed to it. Inside the post-recovery replay window an older
|
||||
/// notification is dropped, and so is one carrying the exact source time already accepted (OPC UA moves the
|
||||
/// source timestamp on every value OR status change, so an equal one is a replay, not a status-only change).
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public async Task After_a_recovery_a_late_older_or_equal_notification_is_dropped()
|
||||
{
|
||||
var (drv, session, clock) = NewConnectedDriver();
|
||||
using var _ = drv;
|
||||
var handle = drv.AddSubscriptionForTest([SpeedRef]);
|
||||
var t0 = clock.Now.UtcDateTime;
|
||||
drv.DeliverNotificationForTest(handle, SpeedRef, Good(668, t0.AddSeconds(2)));
|
||||
SetupReadAt(session, 668, t0.AddSeconds(2));
|
||||
await RecoverInPlaceAsync(drv);
|
||||
var seen = new List<DataChangeEventArgs>();
|
||||
drv.OnDataChange += (_, e) => seen.Add(e);
|
||||
|
||||
drv.DeliverNotificationForTest(handle, SpeedRef, Good(667, t0.AddSeconds(1))); // late replay
|
||||
drv.DeliverNotificationForTest(handle, SpeedRef,
|
||||
new DataValue(new Variant(668), StatusCodes.UncertainLastUsableValue, t0.AddSeconds(2), t0.AddSeconds(2))); // same ts
|
||||
drv.DeliverNotificationForTest(handle, SpeedRef, Good(669, t0.AddSeconds(3)));
|
||||
drv.DeliverNotificationForTest(handle, SpeedRef, Good(665, t0)); // still guarded
|
||||
|
||||
seen.Select(e => e.Snapshot.Value).ShouldBe(new object[] { 669 });
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// #670 F2 re-gate (live gate FAILED on the per-ref disarm): fresh notifications arrived ~1 s after recovery
|
||||
/// and disarmed each ref, then the SDK's late replay arrived 4–6 s later and regressed the node (value 670 with
|
||||
/// a pre-outage source time delivered after newer values). The guard must hold for the WHOLE window against the
|
||||
/// newest ACCEPTED source time per ref — older, equal-to-the-recovery-read and equal-to-the-last-accepted
|
||||
/// replays are all dropped, and counted.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public async Task A_late_replay_after_a_fresh_in_order_notification_is_still_dropped_inside_the_window()
|
||||
{
|
||||
var instance = $"opcua-670-regate-{Guid.NewGuid():N}";
|
||||
using var drops = new OpcUaClientDiagnosticsTests.Capture(instance);
|
||||
var (drv, session, clock) = NewConnectedDriver(instance);
|
||||
using var _ = drv;
|
||||
var handle = drv.AddSubscriptionForTest([SpeedRef]);
|
||||
var t0 = clock.Now.UtcDateTime;
|
||||
drv.DeliverNotificationForTest(handle, SpeedRef, Good(100, t0.AddSeconds(-5)));
|
||||
SetupReadAt(session, 105, t0); // recovery read seeds T0
|
||||
await RecoverInPlaceAsync(drv);
|
||||
var seen = new List<DataChangeEventArgs>();
|
||||
drv.OnDataChange += (_, e) => seen.Add(e);
|
||||
|
||||
clock.Advance(TimeSpan.FromSeconds(1));
|
||||
drv.DeliverNotificationForTest(handle, SpeedRef, Good(116, t0.AddSeconds(6))); // fresh, in order: T1
|
||||
clock.Advance(TimeSpan.FromSeconds(5));
|
||||
drv.DeliverNotificationForTest(handle, SpeedRef, Good(109, t0.AddSeconds(3))); // late replay, older than T1
|
||||
drv.DeliverNotificationForTest(handle, SpeedRef, Good(105, t0)); // replay == T0 (recovery read)
|
||||
drv.DeliverNotificationForTest(handle, SpeedRef, Good(116, t0.AddSeconds(6))); // replay == T1
|
||||
drv.DeliverNotificationForTest(handle, SpeedRef, Good(670, t0.AddSeconds(-10))); // pre-outage replay
|
||||
drv.DeliverNotificationForTest(handle, SpeedRef, Good(117, t0.AddSeconds(7))); // fresh again
|
||||
|
||||
seen.Select(e => e.Snapshot.Value).ShouldBe(new object[] { 116, 117 });
|
||||
drops[OpcUaClientDiagnostics.StaleNotificationDropped].ShouldBe(4);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The re-gated guard still expires: after the window an older source time (an upstream clock step-back) is
|
||||
/// accepted, even though the ref accepted newer notifications inside the window.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public async Task After_the_window_a_clock_step_back_is_accepted_even_after_in_window_notifications()
|
||||
{
|
||||
var (drv, session, clock) = NewConnectedDriver();
|
||||
using var _ = drv;
|
||||
var handle = drv.AddSubscriptionForTest([SpeedRef]);
|
||||
var t0 = clock.Now.UtcDateTime;
|
||||
drv.DeliverNotificationForTest(handle, SpeedRef, Good(1, t0.AddSeconds(-5)));
|
||||
SetupReadAt(session, 2, t0);
|
||||
await RecoverInPlaceAsync(drv);
|
||||
var seen = new List<DataChangeEventArgs>();
|
||||
drv.OnDataChange += (_, e) => seen.Add(e);
|
||||
|
||||
drv.DeliverNotificationForTest(handle, SpeedRef, Good(3, t0.AddSeconds(20)));
|
||||
clock.Advance(TimeSpan.FromSeconds(31));
|
||||
drv.DeliverNotificationForTest(handle, SpeedRef, Good(4, t0.AddSeconds(1))); // clock stepped back
|
||||
drv.DeliverNotificationForTest(handle, SpeedRef, Good(5, t0)); // and again
|
||||
|
||||
seen.Select(e => e.Snapshot.Value).ShouldBe(new object[] { 3, 4, 5 });
|
||||
}
|
||||
|
||||
/// <summary>A notification with no source timestamp passes through the open guard and does not move it.</summary>
|
||||
[Fact]
|
||||
public async Task A_notification_without_a_source_time_passes_the_guard_and_does_not_advance_it()
|
||||
{
|
||||
var (drv, session, clock) = NewConnectedDriver();
|
||||
using var _ = drv;
|
||||
var handle = drv.AddSubscriptionForTest([SpeedRef]);
|
||||
var t0 = clock.Now.UtcDateTime;
|
||||
drv.DeliverNotificationForTest(handle, SpeedRef, Good(1, t0.AddSeconds(-5)));
|
||||
SetupReadAt(session, 2, t0);
|
||||
await RecoverInPlaceAsync(drv);
|
||||
var seen = new List<DataChangeEventArgs>();
|
||||
drv.OnDataChange += (_, e) => seen.Add(e);
|
||||
|
||||
drv.DeliverNotificationForTest(handle, SpeedRef,
|
||||
new DataValue(new Variant(3), StatusCodes.Good, DateTime.MinValue, t0)); // no source time
|
||||
drv.DeliverNotificationForTest(handle, SpeedRef, Good(4, t0.AddSeconds(-1))); // still guarded at T0
|
||||
drv.DeliverNotificationForTest(handle, SpeedRef, Good(5, t0.AddSeconds(1)));
|
||||
|
||||
seen.Select(e => e.Snapshot.Value).ShouldBe(new object[] { 3, 5 });
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Coordinator review of F2: with no recovery in play, an upstream clock that steps BACKWARDS (NTP correction,
|
||||
/// a PLC with a reset clock) must not freeze the node — every notification is published, whatever its time.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void In_steady_state_a_clock_step_backwards_still_publishes()
|
||||
{
|
||||
var (drv, _, clock) = NewConnectedDriver();
|
||||
using var _ = drv;
|
||||
var handle = drv.AddSubscriptionForTest([SpeedRef]);
|
||||
var t0 = clock.Now.UtcDateTime;
|
||||
var seen = new List<DataChangeEventArgs>();
|
||||
drv.OnDataChange += (_, e) => seen.Add(e);
|
||||
|
||||
drv.DeliverNotificationForTest(handle, SpeedRef, Good(1, t0.AddHours(1)));
|
||||
drv.DeliverNotificationForTest(handle, SpeedRef, Good(2, t0)); // clock stepped back an hour
|
||||
drv.DeliverNotificationForTest(handle, SpeedRef, Good(3, t0.AddSeconds(1)));
|
||||
|
||||
seen.Select(e => e.Snapshot.Value).ShouldBe(new object[] { 1, 2, 3 });
|
||||
}
|
||||
|
||||
/// <summary>The replay guard is bounded in time: once its window has passed, an older notification is accepted
|
||||
/// even for a ref that never saw an in-order one.</summary>
|
||||
[Fact]
|
||||
public async Task The_replay_guard_expires()
|
||||
{
|
||||
var (drv, session, clock) = NewConnectedDriver();
|
||||
using var _ = drv;
|
||||
var handle = drv.AddSubscriptionForTest([SpeedRef]);
|
||||
var t0 = clock.Now.UtcDateTime;
|
||||
drv.DeliverNotificationForTest(handle, SpeedRef, Good(10, t0.AddSeconds(5)));
|
||||
SetupReadAt(session, 10, t0.AddSeconds(5));
|
||||
await RecoverInPlaceAsync(drv);
|
||||
var seen = new List<DataChangeEventArgs>();
|
||||
drv.OnDataChange += (_, e) => seen.Add(e);
|
||||
|
||||
clock.Advance(TimeSpan.FromSeconds(29));
|
||||
drv.DeliverNotificationForTest(handle, SpeedRef, Good(9, t0));
|
||||
seen.ShouldBeEmpty("still inside the 30 s window");
|
||||
|
||||
clock.Advance(TimeSpan.FromSeconds(2));
|
||||
drv.DeliverNotificationForTest(handle, SpeedRef, Good(8, t0));
|
||||
seen.ShouldHaveSingleItem().Snapshot.Value.ShouldBe(8);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void The_outage_Bad_does_not_poison_the_ordering_check()
|
||||
{
|
||||
var (drv, _, clock) = NewConnectedDriver();
|
||||
using var _ = drv;
|
||||
var handle = drv.AddSubscriptionForTest([SpeedRef]);
|
||||
var t0 = clock.Now.UtcDateTime;
|
||||
drv.DeliverNotificationForTest(handle, SpeedRef, Good(1, t0));
|
||||
clock.Advance(TimeSpan.FromSeconds(60));
|
||||
drv.RaiseKeepAliveForTest(StatusCodes.BadNoCommunication); // stamped t0+60 s
|
||||
|
||||
var seen = new List<DataChangeEventArgs>();
|
||||
drv.OnDataChange += (_, e) => seen.Add(e);
|
||||
// An upstream sample produced during the outage (server-side) and delivered on recovery: older than the Bad's
|
||||
// detection stamp, newer than the last Good — it is real data and must pass.
|
||||
drv.DeliverNotificationForTest(handle, SpeedRef, Good(2, t0.AddSeconds(30)));
|
||||
|
||||
seen.ShouldHaveSingleItem().Snapshot.Value.ShouldBe(2);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task A_late_notification_older_than_the_recovery_read_is_dropped()
|
||||
{
|
||||
var (drv, session, clock) = NewConnectedDriver();
|
||||
using var _ = drv;
|
||||
var handle = drv.AddSubscriptionForTest([SpeedRef]);
|
||||
var t0 = clock.Now.UtcDateTime;
|
||||
drv.DeliverNotificationForTest(handle, SpeedRef, Good(666, t0));
|
||||
var readAt = t0.AddSeconds(10);
|
||||
session.Setup(s => s.ReadAsync(
|
||||
It.IsAny<RequestHeader>(), It.IsAny<double>(), It.IsAny<TimestampsToReturn>(),
|
||||
It.IsAny<ReadValueIdCollection>(), It.IsAny<CancellationToken>()))
|
||||
.ReturnsAsync(new ReadResponse
|
||||
{
|
||||
ResponseHeader = new ResponseHeader(),
|
||||
Results = [new DataValue(new Variant(670), StatusCodes.Good, readAt, readAt)],
|
||||
DiagnosticInfos = [],
|
||||
});
|
||||
|
||||
drv.RaiseKeepAliveForTest(StatusCodes.BadNoCommunication);
|
||||
drv.RaiseKeepAliveForTest(StatusCodes.Good);
|
||||
await drv.RecoveryTaskForTest;
|
||||
|
||||
var seen = new List<DataChangeEventArgs>();
|
||||
drv.OnDataChange += (_, e) => seen.Add(e);
|
||||
drv.DeliverNotificationForTest(handle, SpeedRef, Good(667, t0.AddSeconds(5))); // replayed, older than the read
|
||||
seen.ShouldBeEmpty();
|
||||
}
|
||||
}
|
||||
+107
@@ -0,0 +1,107 @@
|
||||
using Opc.Ua;
|
||||
using Opc.Ua.Client;
|
||||
using Shouldly;
|
||||
using Xunit;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.OpcUa;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
|
||||
using CoreHistoryReadResult = ZB.MOM.WW.OtOpcUa.Core.Abstractions.HistoryReadResult;
|
||||
using static ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests.ServerEventRig;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests;
|
||||
|
||||
/// <summary>
|
||||
/// Gitea #669 over a real SDK client session: an event HistoryRead capped below the available events pages
|
||||
/// through the PRODUCTION session-bound continuation store (<c>SessionHistoryContinuationStore</c>) — the
|
||||
/// path the session-less unit tests cannot reach — and the union of pages is every event exactly once.
|
||||
/// </summary>
|
||||
public sealed class EventHistoryPagingOverTheWireTests
|
||||
{
|
||||
private static readonly DateTime Epoch = new(2026, 10, 1, 0, 0, 0, DateTimeKind.Utc);
|
||||
|
||||
[Fact]
|
||||
public async Task A_capped_event_read_pages_through_the_session_store_exactly_once()
|
||||
{
|
||||
var ct = TestContext.Current.CancellationToken;
|
||||
var rig = await StartAsync("EvtPagingWire", encrypted: false, ct);
|
||||
try
|
||||
{
|
||||
// Source installed BEFORE the alarm materialises so the folder is promoted with the HistoryRead bit.
|
||||
var events = Enumerable.Range(0, 25)
|
||||
.Select(i => new HistoricalEvent($"e{i:00}", "src", Epoch.AddSeconds(i / 2), Epoch, "m", 500))
|
||||
.ToList(); // pairs share a timestamp, so page boundaries cut through ties
|
||||
rig.Nm.HistorianDataSource = new SqlLikeEventSource(events);
|
||||
rig.MaterialiseNative();
|
||||
rig.Enforce(false, [Key()]);
|
||||
|
||||
using var session = await rig.ConnectAsync(anonymous: true, ct);
|
||||
var node = new NodeId(RawDeviceFolder, (ushort)session.NamespaceUris.GetIndex(V3NodeIds.RawNamespaceUri));
|
||||
|
||||
var collected = new List<string>();
|
||||
byte[]? cp = null;
|
||||
var pages = 0;
|
||||
do
|
||||
{
|
||||
var (ids, next) = await ReadPageAsync(session, node, cp, ct);
|
||||
collected.AddRange(ids);
|
||||
cp = next;
|
||||
(++pages).ShouldBeLessThan(30, "paging must terminate");
|
||||
}
|
||||
while (cp is not null);
|
||||
|
||||
pages.ShouldBeGreaterThan(2, "25 events at a cap of 10 must page over the wire");
|
||||
collected.ShouldBe(events.Select(e => e.EventId));
|
||||
}
|
||||
finally { await rig.DisposeAsync(); }
|
||||
}
|
||||
|
||||
private static async Task<(List<string> Ids, byte[]? Cp)> ReadPageAsync(
|
||||
ISession session, NodeId node, byte[]? cp, CancellationToken ct)
|
||||
{
|
||||
var filter = new EventFilter();
|
||||
filter.SelectClauses.Add(new SimpleAttributeOperand(ObjectTypeIds.BaseEventType, new QualifiedName("EventId"))
|
||||
{
|
||||
AttributeId = Attributes.Value,
|
||||
});
|
||||
var details = new ReadEventDetails
|
||||
{
|
||||
StartTime = Epoch.AddDays(-1),
|
||||
EndTime = Epoch.AddDays(1),
|
||||
NumValuesPerNode = 10,
|
||||
Filter = filter,
|
||||
};
|
||||
var response = await session.HistoryReadAsync(
|
||||
null, new ExtensionObject(details), TimestampsToReturn.Both, false,
|
||||
new HistoryReadValueIdCollection { new HistoryReadValueId { NodeId = node, ContinuationPoint = cp } }, ct);
|
||||
var result = response.Results.ShouldHaveSingleItem();
|
||||
StatusCode.IsBad(result.StatusCode).ShouldBeFalse($"HistoryRead failed: {result.StatusCode}");
|
||||
var he = (HistoryEvent)ExtensionObject.ToEncodeable(result.HistoryData);
|
||||
var ids = he.Events.Select(e => System.Text.Encoding.UTF8.GetString((byte[])e.EventFields[0].Value)).ToList();
|
||||
return (ids, result.ContinuationPoint is { Length: > 0 } c ? c : null);
|
||||
}
|
||||
|
||||
/// <summary>(time, id) order, [start, end), cap + truncation flag — the SQL backends' contract.</summary>
|
||||
private sealed class SqlLikeEventSource(List<HistoricalEvent> events) : IHistorianDataSource
|
||||
{
|
||||
private readonly List<HistoricalEvent> _all =
|
||||
events.OrderBy(e => e.EventTimeUtc).ThenBy(e => e.EventId, StringComparer.Ordinal).ToList();
|
||||
|
||||
public async Task<HistoricalEventsResult> ReadEventsAsync(
|
||||
string? sourceName, DateTime startUtc, DateTime endUtc, int maxEvents, CancellationToken ct)
|
||||
{
|
||||
await Task.Delay(1, ct).ConfigureAwait(false); // complete on a pool thread, like a real backend
|
||||
var window = _all.Where(e => e.EventTimeUtc >= startUtc && e.EventTimeUtc < endUtc).ToList();
|
||||
var cap = maxEvents <= 0 ? 1000 : maxEvents;
|
||||
return new HistoricalEventsResult(window.Take(cap).ToList(), window.Count > cap ? new byte[] { 1 } : null);
|
||||
}
|
||||
|
||||
public Task<CoreHistoryReadResult> ReadRawAsync(string r, DateTime s, DateTime e, uint m, CancellationToken ct) =>
|
||||
NullHistorianDataSource.Instance.ReadRawAsync(r, s, e, m, ct);
|
||||
public Task<CoreHistoryReadResult> ReadProcessedAsync(
|
||||
string r, DateTime s, DateTime e, TimeSpan i, HistoryAggregateType a, CancellationToken ct) =>
|
||||
NullHistorianDataSource.Instance.ReadProcessedAsync(r, s, e, i, a, ct);
|
||||
public Task<CoreHistoryReadResult> ReadAtTimeAsync(string r, IReadOnlyList<DateTime> t, CancellationToken ct) =>
|
||||
NullHistorianDataSource.Instance.ReadAtTimeAsync(r, t, ct);
|
||||
public HistorianHealthSnapshot GetHealthSnapshot() => NullHistorianDataSource.Instance.GetHealthSnapshot();
|
||||
public void Dispose() { }
|
||||
}
|
||||
}
|
||||
+77
-4
@@ -72,6 +72,50 @@ public sealed class HistoryUpdateOverTheWireTests
|
||||
finally { await rig.DisposeAsync(); }
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Part 11 modified-value history over the wire (#668): two inserts and a delete by the key principal, then a
|
||||
/// HistoryReadRaw with <c>IsReadModified = true</c> decodes as a <see cref="HistoryModifiedData"/> whose
|
||||
/// <c>ModificationInfos</c> carry the update type and the principal, and whose values carry the PRIOR value.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public async Task A_modified_read_returns_the_trail_with_principal_and_update_type_over_the_wire()
|
||||
{
|
||||
var ct = TestContext.Current.CancellationToken;
|
||||
var store = new SharedStore();
|
||||
var rig = await StartAsync("HuWireModified", encrypted: false, ct);
|
||||
try
|
||||
{
|
||||
rig.Nm.HistorianDataUpdater = store;
|
||||
rig.Nm.HistorianDataSource = store;
|
||||
rig.Enforce(true, [Key()], ClusterGrant(KeyPrincipal, "acl-editor", Editor));
|
||||
var id = rig.EnsureRawVariable("Speed", "Double", writable: false, Tagname);
|
||||
|
||||
using var session = await rig.ConnectAsync(anonymous: false, ct);
|
||||
var node = RawNode(session, id);
|
||||
await UpdateAsync(session, node, PerformUpdateType.Insert, ct, (T1, 1.5), (T2, 2.5));
|
||||
await DeleteAtTimeAsync(session, node, ct, T1);
|
||||
|
||||
var details = new ReadRawModifiedDetails
|
||||
{
|
||||
StartTime = T1.AddHours(-1), EndTime = T1.AddHours(1), NumValuesPerNode = 100, IsReadModified = true,
|
||||
};
|
||||
var response = await session.HistoryReadAsync(
|
||||
null, new ExtensionObject(details), TimestampsToReturn.Both, false,
|
||||
new HistoryReadValueIdCollection { new HistoryReadValueId { NodeId = node } }, ct);
|
||||
var result = response.Results.ShouldHaveSingleItem();
|
||||
result.StatusCode.Code.ShouldBe(StatusCodes.Good);
|
||||
var data = ExtensionObject.ToEncodeable(result.HistoryData).ShouldBeOfType<HistoryModifiedData>();
|
||||
|
||||
data.DataValues.Select(v => v.SourceTimestamp).ShouldBe([T1, T1, T2]);
|
||||
data.ModificationInfos.Select(m => m.UpdateType)
|
||||
.ShouldBe([HistoryUpdateType.Insert, HistoryUpdateType.Delete, HistoryUpdateType.Insert]);
|
||||
data.ModificationInfos.ShouldAllBe(m => m.UserName == KeyPrincipal);
|
||||
data.DataValues[0].StatusCode.Code.ShouldBe(StatusCodes.BadNoData); // an insert had no prior value
|
||||
((double)data.DataValues[1].Value).ShouldBe(1.5); // the deleted sample's prior value
|
||||
}
|
||||
finally { await rig.DisposeAsync(); }
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// ALLOW beside DENY: the same server, same node — the editor session is admitted, the session
|
||||
/// without the HistoryUpdate grant gets BadUserAccessDenied and the updater is never called.
|
||||
@@ -263,14 +307,21 @@ public sealed class HistoryUpdateOverTheWireTests
|
||||
{
|
||||
private readonly object _gate = new();
|
||||
private readonly SortedDictionary<DateTime, double> _samples = [];
|
||||
private readonly List<HistoryModifiedValue> _trail = [];
|
||||
private int _updateCalls;
|
||||
|
||||
private void Record(DateTime t, double? prior, HistoryModificationType type, HistoryUpdateContext who) =>
|
||||
_trail.Add(new HistoryModifiedValue(
|
||||
new DataValueSnapshot(prior, prior is null ? StatusCodes.BadNoData : StatusCodes.Good, t, t),
|
||||
DateTime.UtcNow, type, who.Principal, _trail.Count + 1, t));
|
||||
|
||||
public int UpdateCalls => Volatile.Read(ref _updateCalls);
|
||||
|
||||
public int Count { get { lock (_gate) return _samples.Count; } }
|
||||
|
||||
public async Task<IReadOnlyList<uint>> UpdateDataAsync(
|
||||
string tag, HistoryUpdateMode mode, IReadOnlyList<HistorizationValue> values, CancellationToken ct)
|
||||
string tag, HistoryUpdateMode mode, IReadOnlyList<HistorizationValue> values, HistoryUpdateContext context,
|
||||
CancellationToken ct)
|
||||
{
|
||||
Interlocked.Increment(ref _updateCalls);
|
||||
await Task.Delay(1, ct).ConfigureAwait(false);
|
||||
@@ -285,21 +336,43 @@ public sealed class HistoryUpdateOverTheWireTests
|
||||
if (_samples.ContainsKey(t)) { codes.Add(StatusCodes.BadEntryExists); continue; }
|
||||
v.Value.TryGetDouble(out var d).ShouldBeTrue();
|
||||
_samples[t] = d;
|
||||
Record(t, null, HistoryModificationType.Insert, context);
|
||||
codes.Add(StatusCodes.GoodEntryInserted);
|
||||
}
|
||||
}
|
||||
return codes;
|
||||
}
|
||||
|
||||
public Task<uint> DeleteRawAsync(string tag, DateTime startUtc, DateTime endUtc, CancellationToken ct) =>
|
||||
public Task<uint> DeleteRawAsync(string tag, DateTime startUtc, DateTime endUtc, HistoryUpdateContext context,
|
||||
CancellationToken ct) =>
|
||||
throw new NotSupportedException();
|
||||
|
||||
public async Task<IReadOnlyList<uint>> DeleteAtTimeAsync(
|
||||
string tag, IReadOnlyList<DateTime> timesUtc, CancellationToken ct)
|
||||
string tag, IReadOnlyList<DateTime> timesUtc, HistoryUpdateContext context, CancellationToken ct)
|
||||
{
|
||||
await Task.Delay(1, ct).ConfigureAwait(false);
|
||||
lock (_gate)
|
||||
return timesUtc.Select(t => _samples.Remove(t) ? StatusCodes.Good : StatusCodes.BadNoEntryExists).ToList();
|
||||
return timesUtc.Select(t =>
|
||||
{
|
||||
if (!_samples.Remove(t, out var prior)) return StatusCodes.BadNoEntryExists;
|
||||
Record(t, prior, HistoryModificationType.Delete, context);
|
||||
return StatusCodes.Good;
|
||||
}).ToList();
|
||||
}
|
||||
|
||||
public async Task<HistoryModifiedReadResult> ReadModifiedAsync(
|
||||
string fullReference, DateTime startUtc, DateTime endUtc, uint maxValues, HistoryModifiedCursor? resumeAfter,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
await Task.Delay(1, cancellationToken).ConfigureAwait(false);
|
||||
fullReference.ShouldBe(Tagname);
|
||||
lock (_gate)
|
||||
{
|
||||
var rows = _trail
|
||||
.Where(m => m.Value.SourceTimestampUtc >= startUtc && m.Value.SourceTimestampUtc < endUtc)
|
||||
.OrderBy(m => m.Value.SourceTimestampUtc).ThenBy(m => m.ModifiedId).ToList();
|
||||
return new HistoryModifiedReadResult(rows, false);
|
||||
}
|
||||
}
|
||||
|
||||
public async Task<CoreHistoryReadResult> ReadRawAsync(
|
||||
|
||||
+101
-27
@@ -6,14 +6,19 @@ using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
|
||||
namespace ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests;
|
||||
|
||||
/// <summary>
|
||||
/// Reconnect + resume leg of the OpcUaClient driver's wire coverage (Gitea #622): the in-process server is
|
||||
/// stopped and a fresh one is booted on the SAME port / application URI / PKI store — a process restart as the
|
||||
/// driver sees it, with an empty address space the test then re-creates. The driver must detect the loss,
|
||||
/// rebuild a session, and its existing subscription must deliver again with no caller action.
|
||||
/// Outage + recovery leg of the OpcUaClient driver's wire coverage (Gitea #622, reworked for #670): the in-process
|
||||
/// server is stopped and a fresh one is booted on the SAME port / application URI / PKI store — a process restart
|
||||
/// as the driver sees it, with an empty address space the test then re-creates. The driver must REPORT the loss
|
||||
/// (subscribed value BadNoCommunication with the value retained, health non-Healthy — what the host's #527
|
||||
/// comms-loss watchdog trips on). Recovery is the watchdog's sequence, replayed here as <c>DriverInstanceActor</c>
|
||||
/// runs it: ShutdownAsync, InitializeAsync, re-subscribe — after which the value comes back Good. There is no
|
||||
/// in-driver reconnect any more (#670: the SDK reconnect handler kept everything Good and Healthy forever).
|
||||
/// </summary>
|
||||
[Trait("Category", "Integration")]
|
||||
public sealed class OpcUaClientDriverReconnectWireTests
|
||||
{
|
||||
private const uint BadNoCommunication = 0x80310000u;
|
||||
|
||||
private static async Task WaitAsync(Func<bool> condition, string because, TimeSpan? timeout = null)
|
||||
{
|
||||
var deadline = DateTime.UtcNow + (timeout ?? TimeSpan.FromSeconds(60));
|
||||
@@ -26,55 +31,124 @@ public sealed class OpcUaClientDriverReconnectWireTests
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task After_a_server_restart_the_driver_reconnects_and_the_subscription_resumes_delivering()
|
||||
public async Task A_server_restart_goes_Bad_then_the_watchdog_reinit_brings_the_value_back_Good()
|
||||
{
|
||||
var ct = TestContext.Current.CancellationToken;
|
||||
var rig = await OpcUaClientDriverWireTests.StartRigAsync("drv-reconnect", ct);
|
||||
await using var _rig = rig;
|
||||
OpcUaClientDriverWireTests.Publish(rig, OpcUaClientDriverWireTests.SpeedId, 10.0f);
|
||||
|
||||
await using var drv = OpcUaClientDriverWireTests.NewDriver(
|
||||
rig, "wire-reconnect", reconnectPeriod: TimeSpan.FromSeconds(1));
|
||||
await using var drv = OpcUaClientDriverWireTests.NewDriver(rig, "wire-reconnect");
|
||||
await drv.InitializeAsync("{}", ct);
|
||||
|
||||
var seen = new List<float>();
|
||||
var seen = new List<(uint Status, object? Value)>();
|
||||
drv.OnDataChange += (_, e) =>
|
||||
{
|
||||
if (e.FullReference == OpcUaClientDriverWireTests.SpeedRef && e.Snapshot.StatusCode == 0u)
|
||||
lock (seen) seen.Add(Convert.ToSingle(e.Snapshot.Value));
|
||||
if (e.FullReference == OpcUaClientDriverWireTests.SpeedRef)
|
||||
lock (seen) seen.Add((e.Snapshot.StatusCode, e.Snapshot.Value));
|
||||
};
|
||||
var states = new List<HostState>();
|
||||
drv.OnHostStatusChanged += (_, e) => { lock (states) states.Add(e.NewState); };
|
||||
|
||||
await drv.SubscribeAsync([OpcUaClientDriverWireTests.SpeedRef], TimeSpan.FromMilliseconds(100), ct);
|
||||
await WaitAsync(() => { lock (seen) return seen.Contains(10.0f); }, "initial value must arrive before the outage");
|
||||
await WaitAsync(() => { lock (seen) return seen.Any(s => s is (0u, 10.0f)); },
|
||||
"initial value must arrive before the outage");
|
||||
|
||||
// Outage: the loss must be DETECTED (host leaves Running) while the server is genuinely down.
|
||||
// Outage: the loss must be REPORTED while the server is genuinely down — Bad on the value with the value
|
||||
// retained, and a health state the watchdog can act on.
|
||||
await rig.RestartAsync(ct, whileDown: async () =>
|
||||
{
|
||||
await WaitAsync(
|
||||
() => { lock (states) return states.Any(s => s != HostState.Running); },
|
||||
"the driver must notice the server is gone", TimeSpan.FromSeconds(30));
|
||||
await WaitAsync(() => { lock (seen) return seen.Any(s => s.Status == BadNoCommunication); },
|
||||
"a lost session must publish BadNoCommunication for the subscribed ref", TimeSpan.FromSeconds(30));
|
||||
lock (seen) seen.Single(s => s.Status == BadNoCommunication).Value.ShouldBe(10.0f);
|
||||
drv.GetHealth().State.ShouldNotBe(DriverState.Healthy);
|
||||
drv.GetHostStatuses().Single().State.ShouldNotBe(HostState.Running);
|
||||
|
||||
// The read stamp must stay frozen for the whole outage — it is the watchdog's silence measure.
|
||||
var frozen = drv.GetHealth().LastSuccessfulRead;
|
||||
frozen.ShouldNotBeNull();
|
||||
await Task.Delay(TimeSpan.FromSeconds(3), ct); // three keep-alive intervals
|
||||
drv.GetHealth().LastSuccessfulRead.ShouldBe(frozen);
|
||||
});
|
||||
|
||||
// The new server is empty: redeploy the nodes and publish a value no earlier delivery could have carried.
|
||||
// The restarted server knows nothing of the old session, so the driver stays down on its own …
|
||||
await Task.Delay(TimeSpan.FromSeconds(3), ct);
|
||||
drv.GetHealth().State.ShouldNotBe(DriverState.Healthy, "nothing in the driver may paper over a dead session");
|
||||
lock (seen) seen.Count(s => s.Status == BadNoCommunication).ShouldBe(1, "one Bad per outage, not per keep-alive");
|
||||
|
||||
// … until the host's comms-loss watchdog re-initialises it (DriverInstanceActor: Reconnecting → ShutdownAsync +
|
||||
// InitializeAsync → Connected → re-subscribe). The new server is empty: redeploy the nodes first, holding the
|
||||
// SAME value as before the outage.
|
||||
rig.EnsureRawVariable("Speed", "Float", writable: true);
|
||||
rig.EnsureRawVariable("Counter", "Int32", writable: false);
|
||||
OpcUaClientDriverWireTests.Publish(rig, OpcUaClientDriverWireTests.SpeedId, 77.0f);
|
||||
OpcUaClientDriverWireTests.Publish(rig, OpcUaClientDriverWireTests.SpeedId, 10.0f);
|
||||
|
||||
await WaitAsync(() => drv.GetHostStatuses().Single().State == HostState.Running,
|
||||
"the driver must reconnect after the server is back");
|
||||
await WaitAsync(() => { lock (seen) return seen.Contains(77.0f); },
|
||||
"the pre-outage subscription must resume and deliver the post-restart value");
|
||||
await drv.ShutdownAsync(ct);
|
||||
await drv.InitializeAsync("{}", ct);
|
||||
drv.GetHealth().State.ShouldBe(DriverState.Healthy);
|
||||
int before;
|
||||
lock (seen) before = seen.Count;
|
||||
await drv.SubscribeAsync([OpcUaClientDriverWireTests.SpeedRef], TimeSpan.FromMilliseconds(100), ct);
|
||||
|
||||
// The IDENTICAL value comes back Good — a (value, status) dedup downstream lets it through.
|
||||
await WaitAsync(() => { lock (seen) return seen.Skip(before).Any(s => s is (0u, 10.0f)); },
|
||||
"the re-subscribed ref must come back Good with its value");
|
||||
|
||||
// Ongoing delivery, not just a one-off initial value.
|
||||
OpcUaClientDriverWireTests.Publish(rig, OpcUaClientDriverWireTests.SpeedId, 88.0f);
|
||||
await WaitAsync(() => { lock (seen) return seen.Contains(88.0f); },
|
||||
"a later change must also reach the resumed subscription");
|
||||
await WaitAsync(() => { lock (seen) return seen.Any(s => s is (0u, 88.0f)); },
|
||||
"a later change must also reach the new subscription");
|
||||
drv.GetHostStatuses().Single().State.ShouldBe(HostState.Running);
|
||||
|
||||
// And the data plane works for plain reads on the new session.
|
||||
var back = await drv.ReadAsync([OpcUaClientDriverWireTests.SpeedRef], ct);
|
||||
back[0].StatusCode.ShouldBe(0u);
|
||||
Convert.ToSingle(back[0].Value).ShouldBe(88.0f);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// #670 review M1 — the subscription dies while the SESSION lives (in production: an outage longer than the
|
||||
/// subscription lifetime but shorter than the session timeout; here: the server deletes it). Keep-alives stay
|
||||
/// Good throughout, so a driver that only watched them would end Healthy with fresh read stamps and a frozen
|
||||
/// value forever. It must go Bad, stay non-Healthy with the stamp frozen (so the #527 watchdog fires), and the
|
||||
/// watchdog's re-initialise must bring the value back.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public async Task A_subscription_lost_while_the_session_lives_is_reported_and_recovered_by_reinit()
|
||||
{
|
||||
var ct = TestContext.Current.CancellationToken;
|
||||
await using var rig = await OpcUaClientDriverWireTests.StartRigAsync("drv-sublost", ct);
|
||||
OpcUaClientDriverWireTests.Publish(rig, OpcUaClientDriverWireTests.SpeedId, 5.0f);
|
||||
|
||||
await using var drv = OpcUaClientDriverWireTests.NewDriver(rig, "wire-sublost");
|
||||
await drv.InitializeAsync("{}", ct);
|
||||
var seen = new List<(uint Status, object? Value)>();
|
||||
drv.OnDataChange += (_, e) =>
|
||||
{
|
||||
if (e.FullReference == OpcUaClientDriverWireTests.SpeedRef)
|
||||
lock (seen) seen.Add((e.Snapshot.StatusCode, e.Snapshot.Value));
|
||||
};
|
||||
await drv.SubscribeAsync([OpcUaClientDriverWireTests.SpeedRef], TimeSpan.FromMilliseconds(100), ct);
|
||||
await WaitAsync(() => { lock (seen) return seen.Any(s => s is (0u, 5.0f)); }, "initial value");
|
||||
|
||||
// Delete the subscription on the server behind the SDK's back; the session itself stays healthy.
|
||||
var session = drv.Session.ShouldNotBeNull();
|
||||
var subId = session.Subscriptions.Single(s => s.Created).Id;
|
||||
await session.DeleteSubscriptionsAsync(null, [subId], ct);
|
||||
|
||||
await WaitAsync(() => { lock (seen) return seen.Any(s => s.Status == BadNoCommunication); },
|
||||
"a subscription the server dropped must be reported as comms loss", TimeSpan.FromSeconds(30));
|
||||
drv.GetHealth().State.ShouldNotBe(DriverState.Healthy);
|
||||
var frozen = drv.GetHealth().LastSuccessfulRead;
|
||||
await Task.Delay(TimeSpan.FromSeconds(3), ct); // good keep-alives keep arriving …
|
||||
drv.GetHealth().State.ShouldNotBe(DriverState.Healthy, "… and must not revive a driver whose data plane is dead");
|
||||
drv.GetHealth().LastSuccessfulRead.ShouldBe(frozen);
|
||||
drv.GetHostStatuses().Single().State.ShouldBe(HostState.Running, "the session itself is fine");
|
||||
|
||||
// The watchdog's re-initialise recreates the subscription and the value comes back Good.
|
||||
await drv.ShutdownAsync(ct);
|
||||
await drv.InitializeAsync("{}", ct);
|
||||
int before;
|
||||
lock (seen) before = seen.Count;
|
||||
await drv.SubscribeAsync([OpcUaClientDriverWireTests.SpeedRef], TimeSpan.FromMilliseconds(100), ct);
|
||||
await WaitAsync(() => { lock (seen) return seen.Skip(before).Any(s => s is (0u, 5.0f)); },
|
||||
"the re-created subscription must deliver the value Good");
|
||||
drv.GetHealth().State.ShouldBe(DriverState.Healthy);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -463,6 +463,56 @@ public sealed class HistoryUpdateTests : IDisposable
|
||||
finally { await host.DisposeAsync(); }
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// #668: the session's API-key principal and key id reach EVERY updater call (they become the Part 11
|
||||
/// ModificationInfo.UserName of the trail rows the backend records).
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public async Task The_key_principal_and_key_id_are_threaded_to_every_updater_call()
|
||||
{
|
||||
var (host, server) = await BootAsync();
|
||||
try
|
||||
{
|
||||
var nm = server.NodeManager!;
|
||||
var updater = new FakeUpdater();
|
||||
nm.HistorianDataUpdater = updater;
|
||||
WireAcl(nm, enabled: false, NodePermissions.HistoryUpdate);
|
||||
var nodeId = Historized(nm, AddressSpaceRealm.Raw, RawPath);
|
||||
var identity = KeyIdentity(OpcUaDataPlaneRoles.HistoryUpdate);
|
||||
|
||||
Update(nm, identity, nodeId, PerformUpdateType.Update, Value(T1, 1d));
|
||||
Call(nm, identity, typeof(DeleteRawModifiedDetails),
|
||||
new DeleteRawModifiedDetails { NodeId = nodeId, StartTime = T1, EndTime = T2, IsDeleteModified = false });
|
||||
Call(nm, identity, typeof(DeleteAtTimeDetails),
|
||||
new DeleteAtTimeDetails { NodeId = nodeId, ReqTimes = new DateTimeCollection { T1 } });
|
||||
|
||||
updater.Callers.Count.ShouldBe(3);
|
||||
updater.Callers.ShouldAllBe(c => c == new HistoryUpdateContext(Principal, KeyId));
|
||||
}
|
||||
finally { await host.DisposeAsync(); }
|
||||
}
|
||||
|
||||
/// <summary>#668: an admitted anonymous session is recorded as the reserved <c>anonymous</c> principal, no key id.</summary>
|
||||
[Fact]
|
||||
public async Task An_anonymous_session_is_recorded_as_the_anonymous_principal()
|
||||
{
|
||||
var (host, server) = await BootAsync();
|
||||
try
|
||||
{
|
||||
var nm = server.NodeManager!;
|
||||
var updater = new FakeUpdater();
|
||||
nm.HistorianDataUpdater = updater;
|
||||
WireAcl(nm, enabled: false, NodePermissions.HistoryUpdate);
|
||||
var nodeId = Historized(nm, AddressSpaceRealm.Raw, RawPath);
|
||||
|
||||
var (_, errors) = Insert(nm, AnonymousIdentity(OpcUaDataPlaneRoles.HistoryUpdate), nodeId, Value(T1, 1d));
|
||||
|
||||
errors[0].StatusCode.Code.ShouldBe(StatusCodes.Good);
|
||||
updater.Callers.ShouldHaveSingleItem().ShouldBe(new HistoryUpdateContext(OpcUaPrincipals.Anonymous, null));
|
||||
}
|
||||
finally { await host.DisposeAsync(); }
|
||||
}
|
||||
|
||||
/// <summary>DeleteAtTime returns the updater's per-time codes in order.</summary>
|
||||
[Fact]
|
||||
public async Task Delete_at_time_passes_per_time_codes_through()
|
||||
@@ -593,6 +643,7 @@ public sealed class HistoryUpdateTests : IDisposable
|
||||
private sealed class FakeUpdater : IHistorianDataUpdater
|
||||
{
|
||||
public List<(string Tag, HistoryUpdateMode Mode, IReadOnlyList<HistorizationValue> Values)> UpdateCalls { get; } = [];
|
||||
public List<HistoryUpdateContext> Callers { get; } = [];
|
||||
public List<(string Tag, DateTime Start, DateTime End)> DeleteRawCalls { get; } = [];
|
||||
public List<IReadOnlyList<DateTime>> DeleteAtTimeCalls { get; } = [];
|
||||
public uint[]? UpdateCodes { get; init; }
|
||||
@@ -601,26 +652,28 @@ public sealed class HistoryUpdateTests : IDisposable
|
||||
public Exception? Throw { get; init; }
|
||||
|
||||
public async Task<IReadOnlyList<uint>> UpdateDataAsync(
|
||||
string tag, HistoryUpdateMode mode, IReadOnlyList<HistorizationValue> values, CancellationToken ct)
|
||||
string tag, HistoryUpdateMode mode, IReadOnlyList<HistorizationValue> values, HistoryUpdateContext context,
|
||||
CancellationToken ct)
|
||||
{
|
||||
await Task.Delay(1, ct).ConfigureAwait(false);
|
||||
if (Throw is not null) throw Throw;
|
||||
lock (UpdateCalls) UpdateCalls.Add((tag, mode, values.ToList()));
|
||||
lock (UpdateCalls) { UpdateCalls.Add((tag, mode, values.ToList())); Callers.Add(context); }
|
||||
return UpdateCodes ?? Enumerable.Repeat(StatusCodes.GoodEntryInserted, values.Count).ToArray();
|
||||
}
|
||||
|
||||
public async Task<uint> DeleteRawAsync(string tag, DateTime startUtc, DateTime endUtc, CancellationToken ct)
|
||||
public async Task<uint> DeleteRawAsync(string tag, DateTime startUtc, DateTime endUtc, HistoryUpdateContext context,
|
||||
CancellationToken ct)
|
||||
{
|
||||
await Task.Delay(1, ct).ConfigureAwait(false);
|
||||
lock (DeleteRawCalls) DeleteRawCalls.Add((tag, startUtc, endUtc));
|
||||
lock (DeleteRawCalls) { DeleteRawCalls.Add((tag, startUtc, endUtc)); lock (UpdateCalls) Callers.Add(context); }
|
||||
return DeleteRawCode;
|
||||
}
|
||||
|
||||
public async Task<IReadOnlyList<uint>> DeleteAtTimeAsync(
|
||||
string tag, IReadOnlyList<DateTime> timesUtc, CancellationToken ct)
|
||||
string tag, IReadOnlyList<DateTime> timesUtc, HistoryUpdateContext context, CancellationToken ct)
|
||||
{
|
||||
await Task.Delay(1, ct).ConfigureAwait(false);
|
||||
lock (DeleteAtTimeCalls) DeleteAtTimeCalls.Add(timesUtc.ToList());
|
||||
lock (DeleteAtTimeCalls) { DeleteAtTimeCalls.Add(timesUtc.ToList()); lock (UpdateCalls) Callers.Add(context); }
|
||||
return DeleteAtTimeCodes ?? Enumerable.Repeat(StatusCodes.Good, timesUtc.Count).ToArray();
|
||||
}
|
||||
}
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user