dohertj2/lmxopcua
1
0
Fork 0
Code Issues 32 Pull Requests Actions Packages Projects Releases Wiki Activity

Merge feat/questdb-historian: QuestDB historian backend (#666)
v2-ci / build (push) Successful in 5m6s
Details
v2-ci / unit-tests (push) Failing after 23m29s
Details

Browse Source 
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
This commit is contained in:
Joseph Doherty
2026-10-11 04:40:42 -04:00
parent 8ec7a83c83 1569a59ad2
commit ee67b6370a
127 changed files with 10633 additions and 1448 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CLAUDE.md
+3 -2
View File
@@ -1325,7 +1325,7 @@ are no longer supported (pinned by a test). Condition teardown must forget the w
## Historian / HistoryRead
**Backend: pluggable — `ServerHistorian:Backend` = `Gateway` (default) or `SqlServer`** (see the subsection below).
**Backend: pluggable — `ServerHistorian:Backend` = `Gateway` (default), `SqlServer` or `QuestDb`** (see the subsection below).
As of the gateway-integration cutover, the **Gateway** backend's historian read, alarm-write, and
continuous-historization paths are all served by the **`ZB.MOM.WW.HistorianGateway`** sidecar, consumed as the Gitea-feed
**`ZB.MOM.WW.HistorianGateway.Client`** gRPC package (`historian_gateway.v1`) behind a thin
@@ -1340,8 +1340,9 @@ historian tagname, which is the tag's driver `FullName`.
### Pluggable backends, SQL Server and HistoryUpdate (2026-10)
- **Selection:** `ServerHistorian:Backend` (`Gateway` default | `SqlServer`; unknown fails host start). `HistorianBackendRegistry` → `IHistorianBackendProvider` builds all five seams; the gateway keys are only required for `Gateway`. `SqlServer` uses `ServerHistorian:SqlServer:{ConnectionString (required), Schema=hist, CommandTimeout=30s, AutoCreateSchema=true}`.
- **Selection:** `ServerHistorian:Backend` (`Gateway` default | `SqlServer` | `QuestDb`; unknown fails host start). `HistorianBackendRegistry` → `IHistorianBackendProvider` builds all five seams; the gateway keys are only required for `Gateway`. `SqlServer` uses `ServerHistorian:SqlServer:{ConnectionString (required), Schema=hist, CommandTimeout=30s, AutoCreateSchema=true}`.
- **SQL Server** (`Driver.Historian.SqlServer`): `hist.Sample` PK `(TagId, SourceTimeUtc)` — writes are insert-if-absent, so outbox redelivery is idempotent. Stores every value kind with its **real status**; aggregates (13, simple per-sample definitions, not Part 13) are pushed down to T-SQL with bucket math in **100 ns ticks**. Docker-dev recipe, schema and aggregate semantics: `docs/Historian.md` §"SQL Server backend". Integration tests need `HISTORIAN_SQL_TEST_CONNECTION`.
- **QuestDB** (`Driver.Historian.QuestDb`, `ServerHistorian:QuestDb:*`, Npgsql over PG wire; docker-dev overlay `docker-compose.questdb.yml`): same kinds/status/aggregates/HistoryUpdate as SqlServer, soft delete by tombstone (INSERT only — never `UPDATE`/`DELETE`; every read filters `NOT deleted`). ⚠️ **Timestamps are µs** (`QuestDbConnection.Round`/`BindTimestamp`; `IHistorianDataSource.TimestampResolutionTicks` = 10 steps HistoryRead paging), **pre-1970 cannot be stored** (writer drops, HistoryUpdate `BadOutOfRange`), every write waits for WAL visibility, TTL is enforced on write commits only, set Npgsql `Maximum Pool Size` < QuestDB's 64. `docs/Historian.md` §"QuestDB backend".
- **Real status fix:** the recorder used to stamp every sample Good (`192`); it now carries the driver's status. ⚠️ On the **Gateway** backend this changes nothing stored: the gateway discards quality (C-002), so AVEVA still stores Good.
- **Comms loss is historized as an outage marker** (`Empty` + `BadNoCommunication`, one per tag per outage) on SqlServer; dropped (`DroppedUnsupportedKind`) on Gateway. Outbox records are **format v2** — drain or delete `ContinuousHistorization:OutboxPath` before downgrading a node.
- **HistoryUpdate** (SqlServer only; Gateway ⇒ `BadHistoryOperationUnsupported`): Insert/Replace/Update/DeleteRaw/DeleteAtTime, gated in both ACL modes — flag off needs the `HistoryUpdate` role from a **Cluster-scoped** grant (+ key re-check), flag on uses `DataPlaneAuthorizer`'s HistoryUpdate operation; `WriteOperate` never implies it. The `HistoryWrite` AccessLevel bit is set only when an updater is wired at materialise time. `docs/Historian.md` §HistoryUpdate, `docs/security.md`.
ZB.MOM.WW.OtOpcUa.slnx
+5
View File
@@ -27,6 +27,8 @@
<Project Path="src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Browser/ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Browser.csproj" />
<Project Path="src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Contracts/ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Contracts.csproj" />
<Project Path="src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.csproj" />
<Project Path="src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.csproj" />
<Project Path="src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.csproj" />
<Project Path="src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.csproj" />
<Project Path="src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Modbus/ZB.MOM.WW.OtOpcUa.Driver.Modbus.csproj" />
<Project Path="src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Modbus.Addressing/ZB.MOM.WW.OtOpcUa.Driver.Modbus.Addressing.csproj" />
@@ -101,6 +103,9 @@
<Project Path="tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Tests/ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Tests.csproj" />
<Project Path="tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Browser.Tests/ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Browser.Tests.csproj" />
<Project Path="tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.Tests/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.Tests.csproj" />
<Project Path="tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests.csproj" />
<Project Path="tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests.csproj" />
<Project Path="tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests.csproj" />
<Project Path="tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.Tests/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.Tests.csproj" />
<Project Path="tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests.csproj" />
<Project Path="tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Modbus.Tests/ZB.MOM.WW.OtOpcUa.Driver.Modbus.Tests.csproj" />
docker-dev/README.md
+27
View File
@@ -160,6 +160,33 @@ and the script fails loudly saying so. A plain `docker compose up -d` also hides
rig without the gate helping: `up -d` blocks on the `depends_on: service_healthy` conditions and
returns ~30 s *after* Traefik starts, by which time the router table is long loaded.
## QuestDB historian overlay
`docker-compose.questdb.yml` is an **optional overlay** (not part of the default rig) that adds one service, `questdb`
(`questdb/questdb:10.0.1`, container `otopcua-dev-questdb-1`, named volume `otopcua-questdb-data`), for the `ServerHistorian:Backend=QuestDb`
historian ([`docs/Historian.md`](../docs/Historian.md#questdb-backend)). Host ports: PG wire **`18812`** (→ 8812) and the web console / REST **`19000`**
(→ 9000), chosen so they do not collide with AdminUI `:9000`/`:9200`.
```bash
# Bring up ONLY questdb (--no-deps: nothing else is touched):
docker compose -f docker-dev/docker-compose.yml -f docker-dev/docker-compose.questdb.yml up -d --no-deps questdb
# Point the central nodes at it (the QuestDb connection string already defaults to the `questdb` service;
# override with OTOPCUA_HISTORIAN_QUESTDB). Keep the overlay -f flag on every compose command for these nodes:
OTOPCUA_HISTORIAN_ENABLED=true OTOPCUA_HISTORIAN_BACKEND=QuestDb OTOPCUA_CONTHIST_ENABLED=true \
docker compose -f docker-dev/docker-compose.yml -f docker-dev/docker-compose.questdb.yml up -d central-1 central-2
# Inspect from the host (REST, no client needed):
curl -G localhost:19000/exec --data-urlencode "query=SELECT tag, count() n FROM hist_sample WHERE NOT deleted GROUP BY tag"
# Integration tests (skipped when the variable is absent):
export HISTORIAN_QUESTDB_TEST_CONNECTION='Host=localhost;Port=18812;Username=admin;Password=quest;Database=qdb;ServerCompatibilityMode=NoTypeLoading'
dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests
```
Tear down with `docker compose -f docker-dev/docker-compose.yml -f docker-dev/docker-compose.questdb.yml rm -sf questdb` (add `docker volume rm otopcua-questdb-data`
to drop the data). The `admin`/`quest` credentials are QuestDB's stock defaults, not secrets.
## Tear down
```bash
docker-dev/docker-compose.questdb.yml
+28
View File
@@ -0,0 +1,28 @@
# docker-dev/docker-compose.questdb.yml — optional QuestDB fixture for the QuestDB historian backend (#666).
#
# Up (touches ONLY the questdb service; it has no depends_on, and --no-deps makes that explicit):
# docker compose -f docker-dev/docker-compose.yml -f docker-dev/docker-compose.questdb.yml up -d --no-deps questdb
#
# Image pinned by the T0 spike (docs/plans/2026-10-10-questdb-spike.md): questdb/questdb:10.0.1,
# pushed to Docker Hub 2026-08-24 (≥ 2 weeks before 2026-10-10; ≥ 8.2.3, the first release with table TTL).
# The image ships /usr/bin/curl, so the healthcheck uses it against the min-HTTP health endpoint on :9003.
#
# PG wire -> localhost:18812 (Host=localhost;Port=18812;Username=admin;Password=quest;Database=qdb;...)
# Console -> http://localhost:19000
services:
questdb:
image: questdb/questdb:10.0.1
container_name: otopcua-dev-questdb-1
ports:
- "18812:8812" # PG wire
- "19000:9000" # web console / REST
volumes:
- otopcua-questdb-data:/var/lib/questdb
healthcheck:
test: ["CMD-SHELL", "curl -fs http://localhost:9003/status || exit 1"]
interval: 10s
timeout: 5s
retries: 12
volumes:
otopcua-questdb-data:
name: otopcua-questdb-data
docker-dev/docker-compose.yml
+10
View File
@@ -441,6 +441,11 @@ services:
# docker compose -f docker-dev/docker-compose.yml up -d central-1 central-2
ServerHistorian__Backend: "${OTOPCUA_HISTORIAN_BACKEND:-Gateway}"
ServerHistorian__SqlServer__ConnectionString: "${OTOPCUA_HISTORIAN_SQL:-Server=sql,1433;Database=OtOpcUaHistorian;User Id=sa;Password=OtOpcUa!Dev123;TrustServerCertificate=True;Encrypt=False}"
# QuestDb = the QuestDB historian; the `questdb` service comes from the docker-dev/docker-compose.questdb.yml overlay
# (add `-f docker-dev/docker-compose.questdb.yml` and OTOPCUA_HISTORIAN_BACKEND=QuestDb).
ServerHistorian__QuestDb__ConnectionString: "${OTOPCUA_HISTORIAN_QUESTDB:-Host=questdb;Port=8812;Username=admin;Password=quest;Database=qdb;ServerCompatibilityMode=NoTypeLoading}"
# Empty = no TTL (data kept forever); a positive day count sets the sample/event table TTL.
ServerHistorian__QuestDb__RetentionDays: "${OTOPCUA_HISTORIAN_QUESTDB_RETENTION_DAYS:-}"
# Continuous historization of driver tag values (Gitea #491's live gate). Gated on BOTH
# ContinuousHistorization:Enabled AND ServerHistorian:Enabled — with either off no recorder is
# spawned, so this stays inert on the default rig. OutboxPath is the FasterLog crash boundary and
@@ -574,6 +579,11 @@ services:
# docker compose -f docker-dev/docker-compose.yml up -d central-1 central-2
ServerHistorian__Backend: "${OTOPCUA_HISTORIAN_BACKEND:-Gateway}"
ServerHistorian__SqlServer__ConnectionString: "${OTOPCUA_HISTORIAN_SQL:-Server=sql,1433;Database=OtOpcUaHistorian;User Id=sa;Password=OtOpcUa!Dev123;TrustServerCertificate=True;Encrypt=False}"
# QuestDb = the QuestDB historian; the `questdb` service comes from the docker-dev/docker-compose.questdb.yml overlay
# (add `-f docker-dev/docker-compose.questdb.yml` and OTOPCUA_HISTORIAN_BACKEND=QuestDb).
ServerHistorian__QuestDb__ConnectionString: "${OTOPCUA_HISTORIAN_QUESTDB:-Host=questdb;Port=8812;Username=admin;Password=quest;Database=qdb;ServerCompatibilityMode=NoTypeLoading}"
# Empty = no TTL (data kept forever); a positive day count sets the sample/event table TTL.
ServerHistorian__QuestDb__RetentionDays: "${OTOPCUA_HISTORIAN_QUESTDB_RETENTION_DAYS:-}"
# Continuous historization — see central-1 for the full note + the gate recipe. Both centrals
# carry it so the pair is symmetric; :9200 round-robins them, and the recorder must exist on
# whichever node holds the drivers.
docs/Configuration.md
+25 -7
View File
@@ -377,11 +377,13 @@ The Galaxy/MxAccess connection settings are **not an `appsettings` section.** Th
> The `OTOPCUA_GALAXY_*` environment variables that v1's in-process `Galaxy.Host` consumed **no longer live in this repo** — they moved into the separately-installed mxaccessgw gateway's own config (see the v1 archive pointer in `docs/README.md` and the Galaxy overview at [`docs/drivers/Galaxy.md`](drivers/Galaxy.md)). The only Galaxy connection secret this repo touches is the gateway API key via `ApiKeySecretRef` above.
### Historian config (`ServerHistorian:Backend` — HistorianGateway or SQL Server)
### Historian config (`ServerHistorian:Backend` — HistorianGateway, SQL Server or QuestDB)
The historian backend is **pluggable**, selected by `ServerHistorian:Backend` (case-insensitive; default `Gateway`;
unknown value ⇒ host start fails listing the allowed values): `Gateway` is the external **`ZB.MOM.WW.HistorianGateway`**
sidecar below, `SqlServer` is a plain SQL Server database (see [`ServerHistorian:SqlServer`](#serverhistoriansqlserver-backendsqlserver)).
sidecar below, `SqlServer` is a plain SQL Server database (see [`ServerHistorian:SqlServer`](#serverhistoriansqlserver-backendsqlserver)),
and `QuestDb` is a QuestDB time-series database reached over the Postgres wire protocol (see
[`ServerHistorian:QuestDb`](#serverhistorianquestdb-backendquestdb)).
The default backend is the gateway, consumed as the
`ZB.MOM.WW.HistorianGateway.Client` gRPC package (the retired Wonderware TCP sidecar is documented at
[`docs/drivers/Historian.Wonderware.md`](drivers/Historian.Wonderware.md)). The OtOpcUa host reads three
@@ -404,15 +406,30 @@ Validated at host start (driver-role nodes) when `ServerHistorian:Enabled` **or*
| Key | Default | Validation / meaning |
|---|---|---|
| `ServerHistorian:Backend` | `Gateway` | `Gateway` or `SqlServer`; anything else fails host start (checked even when the historian is disabled). |
| `ServerHistorian:Backend` | `Gateway` | `Gateway`, `SqlServer` or `QuestDb`; anything else fails host start (checked even when the historian is disabled). |
| `ServerHistorian:SqlServer:ConnectionString` | `""` | **Required.** Historian database connection string (not the ConfigDb). Supply via `${secret:OT-HISTORIAN-SQL}` or env `ServerHistorian__SqlServer__ConnectionString`; the database must already exist. |
| `ServerHistorian:SqlServer:Schema` | `hist` | Must match `^[A-Za-z_][A-Za-z0-9_]{0,127}$` (spliced into DDL/DML). |
| `ServerHistorian:SqlServer:CommandTimeout` | `00:00:30` | Must be `> 0`; rounded up to whole seconds. |
| `ServerHistorian:SqlServer:AutoCreateSchema` | `true` | `true` applies the embedded schema script under an app-lock on first use; `false` only verifies `SchemaVersion >= 1` and fails every historian call (health `LastError`, no crash) if behind. |
HistoryUpdate (SqlServer only) additionally needs `ServerHistorian:Enabled=true` and a key whose principal holds the
`HistoryUpdate` permission — see [`docs/security.md`](security.md#historyupdate-historian-writes). Full semantics, schema, aggregates and the
docker-dev recipe: [`docs/Historian.md`](Historian.md#sql-server-backend).
#### `ServerHistorian:QuestDb` (`Backend=QuestDb`)
Read only when `ServerHistorian:Backend` is `QuestDb`; the gateway keys are then not required. Validated at host start (driver-role nodes)
when `ServerHistorian:Enabled` **or** `AlarmHistorian:Enabled` is true (`QuestDbHistorianOptions.Validate()`).
| Key | Default | Validation / meaning |
|---|---|---|
| `ServerHistorian:QuestDb:ConnectionString` | `""` | **Required.** Npgsql connection string for QuestDB's PG-wire port, e.g. `Host=questdb;Port=8812;Username=admin;Password=quest;Database=qdb;ServerCompatibilityMode=NoTypeLoading`. Env `ServerHistorian__QuestDb__ConnectionString` / `${secret:}` — never commit a real one. When the string does not set Npgsql `Maximum Pool Size`, the backend caps it at **24** (Npgsql's own default of 100 exceeds QuestDB's 64-connection default limit); an explicit value is kept as written, and the effective value is logged at Information. Keep the per-node total of a pair under QuestDB's limit. |
| `ServerHistorian:QuestDb:TablePrefix` | `hist_` | Must match `^[A-Za-z0-9_]+$` (spliced into DDL/DML). Tables are `{prefix}sample`, `{prefix}event`, `{prefix}tag`. |
| `ServerHistorian:QuestDb:RetentionDays` | unset | `>= 1` when set. Applied as table TTL to the sample and event tables (enforced on write commits only — a quiet table never drops). Unset keeps data forever. |
| `ServerHistorian:QuestDb:CommandTimeout` | `00:00:30` | Must be `> 0`; rounded up to whole seconds. |
| `ServerHistorian:QuestDb:AutoCreateSchema` | `true` | `true` creates the tables and reconciles the TTL on first use; `false` only verifies the tables exist and fails every historian call (health `LastError`, no crash) if one is missing. |
| `ServerHistorian:QuestDb:VisibilityTimeout` | `00:00:05` | Must be `>= 0`. How long a write waits for its rows to become visible to queries (WAL apply); `0` disables the wait. A timeout only logs a Warning. |
HistoryUpdate (SqlServer and QuestDb only; QuestDb deletes are soft — tombstones) additionally needs `ServerHistorian:Enabled=true` and a key
whose principal holds the `HistoryUpdate` permission — see [`docs/security.md`](security.md#historyupdate-historian-writes). Full semantics,
schema, aggregates and the docker-dev recipe: [`docs/Historian.md`](Historian.md#sql-server-backend) (SqlServer) and
[`docs/Historian.md`](Historian.md#questdb-backend) (QuestDb: the µs timestamp rules, pre-1970 limits, TTL and the docker-dev overlay).
#### `ContinuousHistorization` (recorder + outbox knobs)
@@ -519,8 +536,9 @@ historian secret this repo reads from the environment is the gateway API key:
| Variable | Effect / default |
|---|---|
| `ServerHistorian__ApiKey` | The HistorianGateway peppered-HMAC key (`histgw_<id>_<secret>`) sent as `Authorization: Bearer`. Supply via environment — **never commit**. Required when `ServerHistorian:Enabled=true` **and `Backend=Gateway`**. |
| `ServerHistorian__Backend` | `Gateway` (default) or `SqlServer`. |
| `ServerHistorian__Backend` | `Gateway` (default), `SqlServer` or `QuestDb`. |
| `ServerHistorian__SqlServer__ConnectionString` | The SQL Server historian connection string. Required when `Backend=SqlServer` and the historian (or alarm history) is enabled. Supply via environment / `${secret:}` — **never commit**. |
| `ServerHistorian__QuestDb__ConnectionString` | The QuestDB historian Npgsql connection string. Required when `Backend=QuestDb` and the historian (or alarm history) is enabled. Supply via environment / `${secret:}` — **never commit**. |
### Driver integration-test / fixture sim endpoints
docs/Historian.md
+218 -21
View File
@@ -11,9 +11,10 @@ Wonderware TCP sidecar backend this replaced was retired — see
[drivers/Historian.Wonderware.md](drivers/Historian.Wonderware.md).)
> **The backend is pluggable (2026-10).** `ServerHistorian:Backend` selects `Gateway` (the default — everything
> described below as "the gateway") or `SqlServer` (a plain SQL Server database the server reads and writes
> itself, which also supports OPC UA **HistoryUpdate**). See [Backend selection](#backend-selection-serverhistorianbackend),
> [SQL Server backend](#sql-server-backend) and [HistoryUpdate](#historyupdate). Wherever this page says
> described below as "the gateway"), `SqlServer` (a plain SQL Server database the server reads and writes
> itself, which also supports OPC UA **HistoryUpdate**) or `QuestDb` (a QuestDB time-series database over the Postgres
> wire protocol; also supports HistoryUpdate, via soft delete). See [Backend selection](#backend-selection-serverhistorianbackend),
> [SQL Server backend](#sql-server-backend), [QuestDB backend](#questdb-backend) and [HistoryUpdate](#historyupdate). Wherever this page says
> "the historian" without qualification it means the selected backend; sections that only apply to one backend say so.
Design references: [docs/plans/2026-06-14-galaxy-phase-c-historian-design.md](plans/2026-06-14-galaxy-phase-c-historian-design.md)
@@ -66,23 +67,23 @@ A tag that is both historized and a native alarm:
(`IHistorianBackendProvider`) that builds all five historian seams — the HistoryRead data source, the continuous-
historization value writer, the alarm-history writer, the tag provisioner and (optionally) the HistoryUpdate
updater. `HistorianBackendRegistry` in the Host owns the name table. An unknown name fails host start with an
`OptionsValidationException` that lists the allowed values (`Gateway`, `SqlServer`), regardless of whether anything
`OptionsValidationException` that lists the allowed values (`Gateway`, `SqlServer`, `QuestDb`), regardless of whether anything
consumes the historian. The existing enable gates (`ServerHistorian:Enabled`, `AlarmHistorian:Enabled`,
`ContinuousHistorization:Enabled`) are unchanged — they simply get their objects from the selected provider.
| | `Gateway` (default) | `SqlServer` |
|---|---|---|
| Storage | AVEVA Historian via the HistorianGateway sidecar | `{Schema}` tables in a SQL Server database you point it at |
| Value kinds recorded | `Double`, `Int64`, `Boolean` (numeric analog only) | all: `Double`, `Int64`, `Boolean`, `String`, `DateTime`, plus `Empty` (a Bad sample with no value) |
| Status code stored | Sent as DA quality but **discarded** by the gateway (C-002); AVEVA stores every sample Good | Stored bit-for-bit (`StatusCode` int column) |
| Comms-loss / driver-Bad publishes | Dropped, counted in `DroppedUnsupportedKind` | Stored as an `Empty` outage-marker sample carrying the Bad code (see [below](#typed-values-real-status-and-outage-markers)) |
| Processed aggregates | Average, Minimum, Maximum, Total, Count | The thirteen in the [aggregate table](#aggregates-processed-reads) |
| HistoryUpdate | No (`BadHistoryOperationUnsupported`) | Yes (Insert / Replace / Update / DeleteRaw / DeleteAtTime) |
| Alarm history | gateway `SendEvent` | `Event` table (insert-if-absent) |
| Provisioning (`EnsureTags`) | numeric types only; others skipped | `MERGE` upsert of every request, all types |
| Connection | `Endpoint` + `ApiKey` + TLS keys (top-level `ServerHistorian:*`) | `ServerHistorian:SqlServer:*` |
| | `Gateway` (default) | `SqlServer` | `QuestDb` |
|---|---|---|---|
| Storage | AVEVA Historian via the HistorianGateway sidecar | `{Schema}` tables in a SQL Server database you point it at | `{prefix}sample` / `event` / `tag` WAL tables in a QuestDB instance |
| Value kinds recorded | `Double`, `Int64`, `Boolean` (numeric analog only) | all: `Double`, `Int64`, `Boolean`, `String`, `DateTime`, plus `Empty` (a Bad sample with no value) | same as `SqlServer` |
| Status code stored | Sent as DA quality but **discarded** by the gateway (C-002); AVEVA stores every sample Good | Stored bit-for-bit (`StatusCode` int column) | Stored bit-for-bit (unsigned, in a `LONG`) |
| Comms-loss / driver-Bad publishes | Dropped, counted in `DroppedUnsupportedKind` | Stored as an `Empty` outage-marker sample carrying the Bad code (see [below](#typed-values-real-status-and-outage-markers)) | Stored as an `Empty` outage-marker sample, as on `SqlServer` |
| Processed aggregates | Average, Minimum, Maximum, Total, Count | The thirteen in the [aggregate table](#aggregates-processed-reads) | The thirteen, pushed down to QuestDB SQL |
| HistoryUpdate | No (`BadHistoryOperationUnsupported`) | Yes (Insert / Replace / Update / DeleteRaw / DeleteAtTime) | Yes — soft delete (tombstones), see [below](#historyupdate-soft-delete) |
| Alarm history | gateway `SendEvent` | `Event` table (insert-if-absent) | `{prefix}event` table (dedup upsert on `(ts, event_id)`) |
| Provisioning (`EnsureTags`) | numeric types only; others skipped | `MERGE` upsert of every request, all types | Upsert into `{prefix}tag`, all types |
| Connection | `Endpoint` + `ApiKey` + TLS keys (top-level `ServerHistorian:*`) | `ServerHistorian:SqlServer:*` | `ServerHistorian:QuestDb:*` |
Recording stays **Primary-only** (#537) and **at-least-once** with the #538 per-tag quarantine on both backends;
Recording stays **Primary-only** (#537) and **at-least-once** with the #538 per-tag quarantine on every backend;
HistoryRead and HistoryUpdate are served from any node (the database is shared). The historian connection string is
**not** a ConfigDb string, and a driver-only node may carry it without breaking the Phase 4 "no ConfigDb" posture.
@@ -114,6 +115,14 @@ and all HistoryRead calls on historized nodes return `GoodNoData` (empty, not an
"Schema": "hist",
"CommandTimeout": "00:00:30",
"AutoCreateSchema": true
},
"QuestDb": {
"ConnectionString": "",
"TablePrefix": "hist_",
"RetentionDays": null,
"CommandTimeout": "00:00:30",
"AutoCreateSchema": true,
"VisibilityTimeout": "00:00:05"
}
}
}
@@ -122,7 +131,7 @@ and all HistoryRead calls on historized nodes return `GoodNoData` (empty, not an
| Key | Type | Default | Description |
|---|---|---|---|
| `Enabled` | bool | `false` | Enable the selected backend's read client. `false` → `NullHistorianDataSource` (empty reads) and no HistoryUpdate. |
| `Backend` | string | `Gateway` | `Gateway` or `SqlServer` (case-insensitive). Unknown ⇒ host start fails. See [Backend selection](#backend-selection-serverhistorianbackend). |
| `Backend` | string | `Gateway` | `Gateway`, `SqlServer` or `QuestDb` (case-insensitive). Unknown ⇒ host start fails. See [Backend selection](#backend-selection-serverhistorianbackend). |
| `Endpoint` | string | `""` | **Gateway backend only.** Absolute gateway URI, e.g. `https://host:5222`. Scheme selects transport (`https://` = TLS, `http://` = h2c plaintext). Required when `Enabled`. |
| `ApiKey` | string | `""` | **Gateway backend only.** The gateway peppered-HMAC key (`histgw_<id>_<secret>`) sent as `Authorization: Bearer`. Required when `Enabled`. **Supply via env `ServerHistorian__ApiKey`.** |
| `UseTls` | bool | `true` | **Gateway backend only.** Connect over TLS; must match the `Endpoint` scheme. |
@@ -143,6 +152,10 @@ and all HistoryRead calls on historized nodes return `GoodNoData` (empty, not an
| `SqlServer:CommandTimeout` | TimeSpan | `00:00:30` | Must be `> 0` | Per-command timeout, rounded up to whole seconds (minimum 1 s). The SQL backend does **not** use `CallTimeout`. |
| `SqlServer:AutoCreateSchema` | bool | `true` | — | `true`: on first use apply any missing embedded script under a `sp_getapplock` (`OtOpcUaHistorianSchema`), so concurrent nodes serialise. `false`: only verify `SchemaVersion >= 1`; if behind, **every** historian call fails with a clear error naming the missing version (surfaced in the read client's health `LastError`, never a host crash). A failed bootstrap is retried on the next call, not latched. |
**`ServerHistorian:QuestDb`** (read only when `Backend=QuestDb`; bound by `HistorianBackendRegistry.BindQuestDb`): six keys — `ConnectionString`
(required), `TablePrefix`, `RetentionDays`, `CommandTimeout`, `AutoCreateSchema`, `VisibilityTimeout`. The full table, defaults and validation
rules are in [QuestDB backend → Configuration](#configuration-serverhistorianquestdb) (and [Configuration.md](Configuration.md#serverhistorianquestdb-backendquestdb)).
**Validation rules** (`ServerHistorianOptionsValidator`, driver-role nodes only; admin-only nodes never build historian singletons):
- `Backend` must name a registered backend — checked **always**, with no I/O, even when nothing is enabled.
@@ -150,6 +163,9 @@ and all HistoryRead calls on historized nodes return `GoodNoData` (empty, not an
(the alarm sink sources its connection from this section even when `ServerHistorian` is off).
- `Backend=Gateway`: `Endpoint` must be an absolute http(s) URI, `ApiKey` non-empty, and `UseTls` must agree with the endpoint scheme.
- `Backend=SqlServer`: the three `SqlServer` rules above. `Endpoint`/`ApiKey`/`UseTls` are **not** required.
- `Backend=QuestDb`: `QuestDbHistorianOptions.Validate()` (connection string required, prefix regex, `RetentionDays >= 1`, `CommandTimeout > 0`,
`VisibilityTimeout >= 0`). `Endpoint`/`ApiKey`/`UseTls` are **not** required. A malformed `TablePrefix` is also rejected by the connection helper
when the provider first builds a seam.
- ⚠️ `Backend=SqlServer` with a malformed `Schema` throws `ArgumentException` while the provider is built, so it stops
host start even when no consumer is enabled (the validator is consumer-gated, provider construction is not).
- The alarm-only warning at startup (`AlarmHistorian` on, `ServerHistorian` off) names the selected backend.
@@ -261,7 +277,7 @@ by `SqlServer`, not by `Gateway`).
(#481) fans `(null, BadNoCommunication)` for every value ref of a comms-lost driver to the dependency mux, and the
recorder cannot tell that from a driver's own Bad publish. So:
- **`SqlServer`:** each disconnect stores **one `Empty` / `BadNoCommunication` (`0x80310000`) sample per historized
- **`SqlServer` and `QuestDb`:** each disconnect stores **one `Empty` / `BadNoCommunication` (`0x80310000`) sample per historized
tag**. A driver's own Bad publish is stored the same way (`Empty` kind plus its own code — e.g. Modbus
`BadCommunicationError` `0x80050000`). This is deliberate: raw reads show the gap, and aggregates and AtTime
interpolation do not bridge it. Cost: one extra row per tag per outage.
@@ -288,7 +304,7 @@ rolling a node back, let the outbox drain to zero (`otopcua.historization.outbox
delete the directory named by `ContinuousHistorization:OutboxPath` (accepting the loss of anything un-drained).
Upgrading needs no action.
**Delivery semantics on `SqlServer`.** Writes are *insert-if-absent* on `(TagId, SourceTimeUtc)`: the outbox's
**Delivery semantics on `SqlServer`** (`QuestDb` is the same, at µs resolution and with tombstones counting as present — see [QuestDB backend](#live-writes-insert-if-absent-tombstones-included)). Writes are *insert-if-absent* on `(TagId, SourceTimeUtc)`: the outbox's
at-least-once redelivery (and both pair nodes writing the same sample in a boot window) converges on one row, and the
first-stored sample wins. **Duplicate timestamps inside one batch collapse to the first.** A `NaN` / `±Infinity` `Double`
cannot be stored in a SQL `float`, so the **value writer stores it as an `Empty` sample** — the tag is not quarantined for
@@ -797,10 +813,190 @@ dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.Integrati
---
## QuestDB backend
Selected with `ServerHistorian:Backend=QuestDb`. Implemented in `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb`
(pure logic shared with the SQL Server backend lives in `Driver.Historian.Sql.Common`). All access is the **Postgres wire protocol via
`Npgsql`** (no ILP, no extra package). Design: [plans/2026-10-10-questdb-historian-design.md](plans/2026-10-10-questdb-historian-design.md);
the spike that pinned every QuestDB-specific form: [plans/2026-10-10-questdb-spike.md](plans/2026-10-10-questdb-spike.md). Pinned image:
`questdb/questdb:10.0.1`.
QuestDB serves every value kind (`Double`, `Int64`, `Boolean`, `String`, `DateTime`, `Empty`), stores the real status, records comms-loss
outage markers, serves the thirteen aggregates and supports **HistoryUpdate** (soft delete), exactly like `SqlServer`. The same conformance
suite (`tests/Drivers/Historian.Conformance`) runs against both backends; the differences below are the only ones.
### Configuration (`ServerHistorian:QuestDb`)
Read only when `Backend=QuestDb`; bound by `HistorianBackendRegistry.BindQuestDb`, validated (consumer-gated, like `SqlServer`) by
`QuestDbHistorianOptions.Validate()`. The gateway keys (`Endpoint`, `ApiKey`, `UseTls`) are not required.
| Key | Default | Validation | Meaning |
|---|---|---|---|
| `QuestDb:ConnectionString` | `""` | **Required** (non-blank) | Npgsql connection string for QuestDB's PG-wire port, e.g. `Host=questdb;Port=8812;Username=admin;Password=quest;Database=qdb;ServerCompatibilityMode=NoTypeLoading`. `ServerCompatibilityMode=NoTypeLoading` skips the `pg_type` load; every query form was verified with it. Supply via env `ServerHistorian__QuestDb__ConnectionString` / `${secret:}` for anything real. |
| `QuestDb:TablePrefix` | `hist_` | `^[A-Za-z0-9_]+$` | QuestDB has no schemas, so the three tables are `{prefix}sample`, `{prefix}event`, `{prefix}tag`. Spliced into DDL/DML, hence the regex. |
| `QuestDb:RetentionDays` | unset | `>= 1` when set | Applied as table TTL to `{prefix}sample` and `{prefix}event` (see [Retention](#retention-ttl)). Unset keeps data forever. |
| `QuestDb:CommandTimeout` | `00:00:30` | `> 0` | Per-command timeout, rounded up to whole seconds (minimum 1 s). |
| `QuestDb:AutoCreateSchema` | `true` | — | `true`: create the tables (and reconcile the TTL) on first use. `false`: only verify the three tables exist; if one is missing every historian call fails with `HistorianSchemaMissingException` (health `LastError`, no crash) and no TTL is touched. A failed bootstrap is not latched: it is re-thrown for 5 s without touching QuestDB, then retried. |
| `QuestDb:VisibilityTimeout` | `00:00:05` | `>= 0` | How long a write waits for its rows to become visible to queries ([WAL visibility](#wal-visibility-and-the-read-your-writes-wait)). `0` disables the wait. |
⚠️ **Connection pool size.** QuestDB's default limit is **64** concurrent PG-wire connections (`pg.net.connection.limit`), while Npgsql's
own default `Maximum Pool Size` is **100**. A busy node (HistoryRead batches, the recorder, the updater and alarm writes all share one pool)
could therefore be refused by QuestDB before Npgsql's pool is full. **When the connection string does not set `Maximum Pool Size`
(or `MaxPoolSize`), the backend now caps it at 24**, so the two nodes of a redundant pair sharing one QuestDB stay under the server limit
together (2 × 24 = 48). An explicit value in the connection string is always kept as written — including one above 64, since QuestDB's
limit is server-configurable. The effective value is logged once at Information when the backend first builds its pool.
### Tables
All three are **WAL tables with `DEDUP UPSERT KEYS`**, so re-sending a row with the same key replaces it (the upsert is what makes
HistoryUpdate Replace work; the writer's own insert-if-absent semantics are built on top, below).
| Table | Key | Partitioning |
|---|---|---|
| `{p}sample` (`ts, tag, kind, num_value, int_value, str_value, status, deleted, server_ts`) | `(ts, tag)` | `PARTITION BY DAY` (TTL when configured) |
| `{p}event` (one-for-one with `hist.Event`) | `(ts, event_id)` | `PARTITION BY DAY` (TTL when configured) |
| `{p}tag` (`ts, name, data_type, eng_unit, description, updated_utc`) | `(ts, name)` with `ts` fixed at `1970-01-01` — an upsert by name | `PARTITION BY YEAR`, never expires |
`status` holds the uint OPC UA status code **unsigned** in a `LONG` (so "Bad" is `status >= 2147483648`, not `< 0`); `int_value` also holds
Booleans as 0/1; `kind` is the `HistorianValueKind` in a `BYTE`. Tag names are limited to 400 characters (longer is a per-tag failure,
never truncated). `{p}tag` is the **only** "is this tag known" authority (`QuestDbTagCatalog`): reads never create a tag, Insert/Update and the
live writer do, Replace and the deletes do not.
### Timestamps: microseconds
QuestDB's designated timestamp is a microsecond. Every inbound `DateTime` is normalised by `QuestDbConnection.Round` (Unspecified ⇒ UTC, Local
⇒ UTC, ticks truncated to a multiple of 10, `MinValue`/`MaxValue` nudged inward because Npgsql would send them as ±infinity) and bound only
through `BindTimestamp`. Consequences:
- Two live samples for one tag **less than 1 µs apart** collapse to one; **the first wins**. A batch is deduplicated client-side after rounding.
- **Raw and Processed results carry stored (µs) timestamps.** A sub-µs request bound is rounded, and the rounded start *is* the request start,
so a Processed bucket's timestamp is the rounded window start (a sub-µs label divergence from `SqlServer`, which labels with the exact
start; values and bucket membership are identical).
- **AtTime** evaluates at the rounded time but **stamps each result with the time the client asked for**, so a client can match answers to
requests. Stored data is µs-only, so a sub-µs request adjacent to a sample is an exact hit.
- `IHistorianDataSource.TimestampResolutionTicks` is **1** by default and **10** on QuestDB. HistoryRead-Raw paging steps off a drained
timestamp by that amount (forward `T + resolution`, reverse `T − resolution`). Stepping by 1 tick would be truncated back to `T` and a
one-value-per-page forward read would loop forever. Same-timestamp ties (legal on the Gateway backend) are still paged within the timestamp.
- **Pre-1970 timestamps cannot be stored** (QuestDB: `designated timestamp before 1970-01-01 is not allowed`; the whole INSERT fails). So:
- the **live writer** drops such samples with at most one Warning per tag per 10 minutes (a device with an unset clock would otherwise warn every drain pass; every dropped sample is still counted in `DroppedPre1970`) — a retry could never succeed, and sending
them would push the tag into the #538 quarantine;
- **HistoryUpdate** Insert / Replace / Update of such an item returns `BadOutOfRange` (nothing is sent); `DeleteAtTime` returns
`BadNoEntryExists`; `DeleteRaw` finds nothing;
- an alarm event with such a time is a `PermanentFail` (dead-letter);
- **read bounds before 1970 work** (a Raw/Processed window may start before the epoch; bounds are never clamped, only `MinValue`/`MaxValue` nudged).
- **`Int64` `long.MinValue`.** QuestDB's `LONG` NULL is `long.MinValue`, so a stored `FromInt64(long.MinValue)` has `int_value` NULL. The read path
repairs it (`kind = Int64 ∧ int_value IS NULL ∧ num_value = −2^63` ⇒ `long.MinValue`); without the repair it would read back as `Empty`/0. Likewise
`NaN`/`±Infinity` doubles read back as NULL, so they are stored as `Empty` + `BadOutOfRange` (live) or rejected `BadInvalidArgument` (HistoryUpdate), as on `SqlServer`.
### Live writes: insert-if-absent, tombstones included
The recorder's writer (`QuestDbHistorianValueWriter`) has the same first-wins redelivery contract as SQL Server:
1. group by tag, round to µs, drop in-batch duplicates (first wins), drop pre-1970 samples;
2. read the timestamps that already exist for the tag (one range query per cluster of nearby times, **`deleted` rows included**);
3. `INSERT` only the absent rows — parameterised multi-row, at most **1,000 rows per statement**;
4. [wait for WAL visibility](#wal-visibility-and-the-read-your-writes-wait).
Because `DEDUP UPSERT` would otherwise let a re-sent sample overwrite the stored one, step 2 is what keeps outbox redelivery idempotent — and
**a live-writer redelivery never resurrects a deleted row**: a tombstone counts as present. Any SQL or transport failure returns `false`
(the recorder retries; the #538 per-tag quarantine still applies); the writer never throws.
### WAL visibility and the read-your-writes wait
QuestDB applies WAL commits asynchronously, so a row just inserted may not be visible to the next query (measured: up to 7 of 20 inserts invisible
to an immediate re-read). **Every write path waits** — the live writer, the HistoryUpdate updater (including between DeleteRaw pages), the tag
catalog/provisioner and the alarm writer (`QuestDbWalVisibility`): after the INSERT(s) it reads `sequencerTxn` from `wal_tables()` and polls
`writerTxn` about every 10 ms until it catches up, the table is reported `suspended` (Warning, stop), or `VisibilityTimeout` elapses (Warning).
A timeout never changes a result code and never discards data; the rows simply become visible shortly after. Measured: median 0 ms, p95 6 ms.
### HistoryUpdate (soft delete)
All five operations are supported, and **every one is an `INSERT`**: QuestDB `UPDATE`/`DELETE` are never issued. "Exists" means a row that is
present **and not deleted**.
| Operation | Effect | Result |
|---|---|---|
| Insert | write the row when none exists — **a tombstone counts as absent, so Insert over a tombstone succeeds** | `GoodEntryInserted` / `BadEntryExists` |
| Replace | overwrite (dedup upsert) when a live row exists | `GoodEntryReplaced` / `BadNoEntryExists` |
| Update | write either way | `GoodEntryReplaced` or `GoodEntryInserted` |
| DeleteAtTime | write a tombstone (`deleted = true`, value NULL, `BadNoData`) where a live row exists | `Good` / `BadNoEntryExists` |
| DeleteRaw | tombstone every live row in `[start, end)`, 1,000 per page, waiting for visibility between pages | the same codes as `SqlServer` |
The per-item codes come from the shared `HistoryUpdatePlanner`, so they equal `SqlServer`'s (including the duplicate-timestamp-in-one-request rules
above). **Every read and aggregate filters `NOT deleted`.** A client Insert at a deleted timestamp re-populates it (the OPC UA intent); only the
*live writer's* redelivery is barred from doing so.
⚠️ **Concurrency.** The updater serializes calls **per tag, in-process only** (64 striped locks held from the existence read through the visibility
wait), so two concurrent Insert/Replace/Delete batches on one tag cannot both act on a stale read. It does **not** lock against the live writer,
nor against a second node. The existence checks read committed data, so a recorder sample at the same timestamp still in flight in the WAL can race
an Insert/Replace (updates normally target past timestamps), and two nodes writing the same *new* key concurrently can produce last-write-wins
(QuestDB has no `UPDLOCK`; the Primary gate makes it rare). This residual is accepted.
### Reads
Every query filters `NOT deleted`; every value goes through `StoredSample`, so the quality clamp equals `SqlServer`'s.
- **Raw:** `ts` range, `ORDER BY ts [DESC] LIMIT $n` (bound, `int`); `ContinuationPoint` is always null — the node manager pages by re-reading
from the last timestamp ± `TimestampResolutionTicks`.
- **AtTime:** previous-or-equal and next neighbour of each time by `LIMIT 1` seeks, `UNION ALL`-ed, **50 times per statement** (QuestDB 10.0.1 refuses
more than 126 branches). `AtTimeEvaluator` decides each value, as on `SqlServer`.
- **Events:** `[start, end)` by `ts`, ordered `(ts, event_id)`, `maxEvents + 1` fetched to produce the continuation point.
- **Processed:** one query per node; bucket index `(cast(ts AS LONG) * 10 - startUnixTicks) / intervalTicks` (never `SAMPLE BY`, whose alignment
differs), so bucket edges are identical to `SqlServer` (start-inclusive, end-exclusive, partial last bucket, interval 0 ⇒ one bucket, 100,000-bucket cap).
Min/max times use `row_number()` windows with the earliest-timestamp tie-break.
### Retention (TTL)
`RetentionDays` becomes `TTL n DAYS` on `{p}sample` and `{p}event` (never on `{p}tag`: TTL must be a multiple of the partition size). QuestDB drops
expired day partitions itself, tombstones included.
- At bootstrap the current TTL is read from `tables()` (`ttlValue`, `ttlUnit`). QuestDB **normalises the unit** (7 days reads `1 WEEK`), so the
bootstrapper compares in **hours** (`HOUR`=1, `DAY`=24, `WEEK`=168; any other unit counts as different) and issues
`ALTER TABLE … SET TTL n DAYS` only on a difference (`SET TTL 0 DAYS` removes it). The `ALTER` is applied through the WAL, so it does not
re-read to verify; a very large `RetentionDays` that QuestDB normalises into MONTH/YEAR is re-`ALTER`ed on every boot (harmless).
- ⚠️ **TTL is enforced only on write commits.** QuestDB checks for expired partitions when a write commits to the table, so **a quiet table never
drops anything**: partitions older than the TTL linger until the next write lands. Do not read a stale-looking partition as a broken TTL.
- Expiry is judged against the wall clock, so a future-dated sample does not expire current data.
- The TTL is visible in table metadata: `SELECT table_name, ttlValue, ttlUnit FROM tables() WHERE starts_with(table_name, 'hist_')`.
### Docker-dev recipe
QuestDB is an **optional compose overlay**, `docker-dev/docker-compose.questdb.yml`, so the default rig is unchanged. See
[`docker-dev/README.md`](../docker-dev/README.md#questdb-historian-overlay).
```bash
# 1. Bring up ONLY the questdb service (PG wire on host :18812, console/REST on host :19000).
docker compose -f docker-dev/docker-compose.yml -f docker-dev/docker-compose.questdb.yml up -d --no-deps questdb
# 2. Recreate the central nodes against it (the QuestDb connection string already defaults to the `questdb` service):
OTOPCUA_HISTORIAN_ENABLED=true OTOPCUA_HISTORIAN_BACKEND=QuestDb OTOPCUA_CONTHIST_ENABLED=true \
docker compose -f docker-dev/docker-compose.yml -f docker-dev/docker-compose.questdb.yml up -d central-1 central-2
# 3. Mark a rig tag historized, deploy, then watch rows land (REST, no client needed):
curl -G localhost:19000/exec --data-urlencode "query=SELECT tag, count() n, max(ts) last FROM hist_sample WHERE NOT deleted GROUP BY tag"
```
Only the Primary of a pair records. Read back with `Client.CLI historyread`, rewrite with `Client.CLI historyupdate`.
### Integration tests
`tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests` (bootstrap incl. two concurrent bootstrappers, TTL via `tables()`, writes,
tombstones and redelivery, the visibility wait, updater, reads, paging, and the shared conformance suite) is gated on
**`HISTORIAN_QUESTDB_TEST_CONNECTION`** — an Npgsql connection string for the PG-wire port. Absent ⇒ every test skips. Each test class uses a
unique throwaway prefix (`t_<guid>_`) and drops its tables afterwards. The QuestDB server allows 64 connections, so the fixture prunes idle pool connections.
```bash
docker compose -f docker-dev/docker-compose.yml -f docker-dev/docker-compose.questdb.yml up -d --no-deps questdb
export HISTORIAN_QUESTDB_TEST_CONNECTION='Host=localhost;Port=18812;Username=admin;Password=quest;Database=qdb;ServerCompatibilityMode=NoTypeLoading'
dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests
```
---
## HistoryUpdate
OPC UA **HistoryUpdate** lets a client insert, replace, update and delete stored raw history. It is available only when the selected
backend provides an updater — **`SqlServer` does; `Gateway` does not**, and an updater is registered only when `ServerHistorian:Enabled=true`.
backend provides an updater — **`SqlServer` and `QuestDb` do; `Gateway` does not**, and an updater is registered only when `ServerHistorian:Enabled=true`.
Updates go straight to the backend (not through the outbox), are **not Primary-gated** (the database is shared), and share the HistoryRead
batch limiter and `HistoryReadDeadline`.
@@ -814,7 +1010,8 @@ batch limiter and `HistoryReadDeadline`.
| `DeleteRawModifiedDetails` (`IsDeleteModified=false`) | `HistoryDeleteRawModified` | one node-level code: `Good`; `BadNoData` if `[start, end)` held nothing |
| `DeleteAtTimeDetails` | `HistoryDeleteAtTime` | per time: `Good`; `BadNoEntryExists` |
Each backend call is one transaction (`MERGE … OUTPUT $action` against the TVP). The node-level `StatusCode` stays `Good` when item results are
Each backend call is applied as a unit (SqlServer: one transaction, `MERGE … OUTPUT $action` against the TVP; QuestDb: an INSERT per
chunk plus the WAL-visibility wait, no transaction — see [QuestDB backend](#questdb-backend)). The node-level `StatusCode` stays `Good` when item results are
mixed (Part 11 reports item outcomes in `OperationResults`); it is Bad only for whole-node failures.
**Per-item rejections** (answered without touching SQL; other items still proceed): a null item ⇒ `BadInvalidArgument`;
docs/plans/2026-10-10-questdb-historian-design.md
+455
View File
@@ -0,0 +1,455 @@
# QuestDB historian backend — design
**Status:** approved section by section on 2026-10-10. Gitea #666. **As-built pass (T8, 2026-10-11):** every section below was re-checked against
the merged code; where the two disagreed the text was corrected inline and marked **(as built)**. User documentation: `docs/Historian.md` §"QuestDB backend".
**Builds on:** `docs/plans/2026-10-09-pluggable-historian-design.md`, which defines the provider seam, typed values, the
HistoryUpdate service and the SQL Server backend, all merged to master as `8ec7a83c`.
## 0. Decisions (user, 2026-10-10)
| Topic | Decision |
|---|---|
| HistoryUpdate deletes | **Soft delete.** Rows carry a `deleted` flag. A delete writes a tombstone at the same `(tag, ts)`. Every read and aggregate excludes tombstones. All five HistoryUpdate operations are supported. |
| Write path | **Postgres wire protocol only, via `Npgsql`.** It is already centrally pinned at 10.0.3. No ILP, no new NuGet package. |
| Retention | **Included.** `RetentionDays` is applied as table TTL. QuestDB drops expired day partitions itself. |
| Approach | **B: shared dialect-free core plus per-backend SQL.** The pure logic moves out of the SQL Server backend into `Driver.Historian.Sql.Common`. One conformance suite runs against both backends. |
| Timestamps | **Microseconds.** Postgres-wire timestamps are microsecond resolution, and real source timestamps are at best milliseconds. Two samples less than 1 µs apart collapse to one, and the first wins — within a batch, and also against an already-stored row (the existence check runs on the rounded time). This is documented. |
Out of scope:
- event HistoryUpdate;
- Part 11 modified values (#668);
- time-weighted aggregates (#671);
- ILP ingest;
- QuestDB authentication beyond the connection string;
- any change to the Gateway backend.
## 1. Components
### 1.1 `ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common` (new, no I/O)
This is extracted from `Driver.Historian.SqlServer` with **no behaviour change**:
- **`StoredSample`.** It holds the time, the value, the stored status. It classifies a sample as
usable, good, bad or uncertain. It applies the quality-contract clamp: a NULL value carrying a non-Bad status is
served as `BadNoData`. It also defines the info-bit masks (`CodeBitsMask`, `SeverityBadBit`, `SeverityMask`).
- **`AtTimeEvaluator`.** This is the former `SqlServerHistorianDataSource.AtTimeValue`. It covers interpolation and
stepping, the outage rule (previous neighbour Bad ⇒ null plus that code; previous usable and next Bad ⇒ hold the
previous value as `UncertainDataSubNormal`), stepping forward after the last sample, `T > now` ⇒ Uncertain, and
masking `& ~0x3` before the info bits are set.
- **`BucketRow` + `AggregateEvaluator`.** This is the former `Evaluate`. Each backend's SQL produces the **same
per-bucket statistics**: total count, usable/good/bad/uncertain counts, numeric sum/min/max, first and last value
and time, the time of the min and of the max, and so on. The shared evaluator maps a `BucketRow` to the OPC UA
value and status for each of the 13 supported aggregates. Unsupported aggregates throw
`HistoryAggregateNotSupportedException`.
- **`BucketPlan`.** It handles window normalisation (a reversed window is swapped and returned descending, as built
today), interval 0 ⇒ one bucket, the 100,000-bucket cap, and tick arithmetic. It exposes the bucket start and width
in **ticks** and in **microseconds**.
- **`HistorianColumnMapping`.** It maps a `HistorianValueKind` to the stored data-type code (the former
`SqlHistorianDataTypes.FromKind`). It also maps a `HistorianValue` to its value columns (num/int/str) and back.
- **`HistoryUpdateResults`.** These are the per-item result rules for Insert, Replace, Update, DeleteAtTime and
DeleteRaw given "did a live row exist". They include the NaN/Inf rule (rejected `BadInvalidArgument` on update,
stored as `BadOutOfRange` Empty on the live write) and the rule rejecting an Empty value with a non-Bad status.
- **Shared constants.** The named status consts that both backends emit (`BadOutOfRange` and others) move into one
parity-visible `*StatusCodes` class here. Backend-specific codes stay with their backend.
`Driver.Historian.SqlServer` then keeps only its T-SQL, the connection, the bootstrapper, the tag catalog and the
seam classes, all calling the shared core.
### 1.2 `ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb` (new)
| Class | Responsibility |
|---|---|
| `QuestDbHistorianOptions` | Binds `ServerHistorian:QuestDb` (§4) and validates it. |
| `QuestDbConnection` | Npgsql `NpgsqlDataSource` (pooled), command timeouts, transient-error classification, UTC timestamp reads. |
| `QuestDbSchemaBootstrapper` | Idempotent `CREATE TABLE IF NOT EXISTS` for the three tables, then applies TTL. It is concurrency-safe across two nodes starting together, and caches a failure for 5 s. |
| `QuestDbHistorianValueWriter` | `IHistorianValueWriter` for live samples (§2.2). |
| `QuestDbTagCatalog` | **(as built)** The only "is this tag known" authority (R7): a `hist_tag` row, positive results cached. Reads never create a tag; the live writer and HistoryUpdate Insert/Update do (Replace and the deletes do not). Re-inserting a known tag is skipped, because the dedup upsert would erase the unit and description the provisioner wrote. |
| `QuestDbWalVisibility` | **(as built)** The read-your-writes wait (§2.3), used by every write path. |
| `QuestDbTimeClusters` | **(as built)** Groups request times into ranges (gap > 1 min starts a new one) so existence reads are one range query per cluster. |
| `QuestDbTables`, `QuestDbAggregateQueries` | **(as built)** DDL / TTL helpers, and the push-down aggregate SQL (§3). |
| `QuestDbTagProvisioner` | `IHistorianProvisioning`: upserts `hist_tag` rows in chunks; **(as built)** a request without a unit or description keeps the stored one. |
| `QuestDbAlarmHistorianWriter` | `IAlarmHistorianWriter`: **(as built)** one INSERT per event (not batched), with the deterministic SHA-256 `event_id`. Outcomes: Ack; `RetryPlease` for transient and unclassified errors; `PermanentFail` only for a pre-1970 time or a server message naming an inconvertible / out-of-range / too-long / null value. |
| `QuestDbHistorianDataSource` | `IHistorianDataSource` (§3), with QuestDB SQL for raw, at-time, events and bucket statistics. |
| `QuestDbHistorianDataUpdater` | `IHistorianDataUpdater` (§2.3), the soft-delete updater. |
| `QuestDbHistorianProvider` | `IHistorianBackendProvider`: `Name = "QuestDb"`, `SupportedKinds` = all six (FrozenSet), one shared connection and bootstrapper. |
Host changes:
- `HistorianBackendRegistry` gains `"QuestDb"`.
- `ServerHistorianOptionsValidator` validates the QuestDb section, gated on consumers, the same way it gates
SqlServer.
- `Host.csproj` gets a ProjectReference.
- **(as built)** `IHistorianDataSource` gains the default member `TimestampResolutionTicks` (1; QuestDB 10), and the node manager's Raw paging steps off a drained timestamp by it (§3).
- **(as built)** The provider builds its shared pieces (connection, bootstrapper, catalog, WAL wait) on the first `Create*` call, not in the constructor, so a provider nobody consumes never fails on its options.
## 2. Schema and writes
### 2.1 Tables
QuestDB has no schemas, so the tables carry a configurable `TablePrefix` (default `hist_`, `[A-Za-z0-9_]+`). All
three are **WAL tables with `DEDUP UPSERT KEYS`**, so re-sending a row with the same key replaces it.
**T0-confirmed** against `questdb/questdb:10.0.1` (`docs/plans/2026-10-10-questdb-spike.md` §DDL, Q3, Q5, Q10). The
`TTL n DAYS` clause goes **between `PARTITION BY` and `WAL`**. Placed after the dedup clause it fails with `unexpected
token [TTL]`. It is emitted only when `RetentionDays` is set, as an inlined integer, because it cannot be a parameter.
```sql
CREATE TABLE IF NOT EXISTS hist_sample (
ts TIMESTAMP, tag SYMBOL, kind BYTE, num_value DOUBLE, int_value LONG, str_value VARCHAR,
status LONG, deleted BOOLEAN, server_ts TIMESTAMP
) TIMESTAMP(ts) PARTITION BY DAY [TTL n DAYS] WAL DEDUP UPSERT KEYS(ts, tag);
CREATE TABLE IF NOT EXISTS hist_event (
ts TIMESTAMP, event_id VARCHAR, source SYMBOL, alarm_id VARCHAR, alarm_name VARCHAR, alarm_type_name VARCHAR,
severity INT, event_kind VARCHAR, message VARCHAR, user_name VARCHAR, comment VARCHAR, received_utc TIMESTAMP
) TIMESTAMP(ts) PARTITION BY DAY [TTL n DAYS] WAL DEDUP UPSERT KEYS(ts, event_id);
CREATE TABLE IF NOT EXISTS hist_tag (
ts TIMESTAMP, name SYMBOL, data_type BYTE, eng_unit VARCHAR, description VARCHAR, updated_utc TIMESTAMP
) TIMESTAMP(ts) PARTITION BY YEAR WAL DEDUP UPSERT KEYS(ts, name); -- ts is always 1970-01-01, which makes this an upsert by name
```
`hist_event` maps `hist.Event` one-for-one:
- `EventTimeUtc` → `ts`, `EventId` → `event_id`, `SourceName` → `source`;
- `AlarmId`/`AlarmName`/`AlarmTypeName` → `alarm_id`/`alarm_name`/`alarm_type_name`;
- `Severity` → `severity`, `EventKind` → `event_kind`, `Message` → `message`;
- `UserName` → `user_name`, `Comment` → `comment`, `ReceivedTimeUtc` → `received_utc`.
`event_id VARCHAR` works as a dedup key (T0 Q3), so re-sending an event replaces it. The S4 fallback (no dedup on
`hist_event`) is not used.
Column details (**T0-confirmed**):
- `status` holds the uint status code **unsigned** in a LONG (0..4294967295, never negative). So "severity Bad" is
`status >= 2147483648`, not `< 0` as in SQL Server's signed `int`.
- `int_value` also holds Booleans as 0/1, matching SQL Server.
- `kind` is `HistorianValueKind` in a `BYTE`.
- Read-back CLR types:
| Column type | Npgsql type name | CLR type / reader call |
|---|---|---|
| `BYTE` | `smallint` | `Int16` (`GetInt16`) |
| `SYMBOL`, `VARCHAR` | `character varying` | `String` |
| `LONG` | `bigint` | `Int64` |
| `INT` | `integer` | `Int32` |
| `BOOLEAN` | `boolean` | `Boolean` |
| `DOUBLE` | `double precision` | `Double` |
| `TIMESTAMP` | `timestamp without time zone` | `DateTime`, **`Kind=Unspecified`**; the reader applies `SpecifyKind(…, Utc)` |
- **NULL semantics:**
- DOUBLE NaN and ±Inf read back as NULL.
- `long.MinValue` (LONG) and `int.MinValue` (INT) are QuestDB's NULL sentinels.
- `BYTE` and `BOOLEAN` have **no NULL**: unset or DBNull reads `0` / `false`. So `deleted` defaults to `false`, and a
LEFT-JOINed "no row" yields `kind = 0`. Absence must be detected on a NULL `ts`, not on `kind`.
- **Ruling RC2 (as built):** an Int64 sample of exactly `long.MinValue` stores `int_value` as NULL. The QuestDB read
path repairs it before the shared `FromColumns`: `kind = Int64 ∧ int_value IS NULL ∧ num_value = -2^63` ⇒
`long.MinValue`. That double is exact. Without the repair the sample would be served as 0.
- A designated timestamp **before 1970-01-01 cannot be inserted** (`designated timestamp before 1970-01-01 is not
allowed`). See §2.2 (RC1).
- Tag names: SYMBOL and VARCHAR take 400 characters (non-ASCII included). SYMBOL grows past its capacity hint
(16,384) with no error, and was tested with 30,000 distinct tags. No `CAPACITY` clause is needed.
### 2.2 Live writes (recorder) — insert only if absent, the same as SQL Server
1. Group the batch by tag and remove duplicates within the batch. **The first value wins.** A timestamp is rounded
to microseconds before this step, so microsecond collisions also collapse.
2. For each tag, fetch the timestamps that already exist. **(as built)** The batch's times are grouped into clusters
(`QuestDbTimeClusters`: a gap over 1 minute starts a new range; the live writer also caps a range at 1,000 times), with one range query per cluster, so one
far-away outlier does not widen a single query over every stored row in between. **Tombstones are included**, so a re-sent
outbox batch can never resurrect a deleted row or overwrite a HistoryUpdate change. T0-confirmed form (Q11): `SELECT ts
FROM {p}sample WHERE tag = $1 AND ts >= $2 AND ts <= $3`, with no `NOT deleted`, followed by a client-side set test. The tag is
registered in `QuestDbTagCatalog` first (a write creates an unknown tag).
3. Insert only the new rows with parameterised multi-row `INSERT … VALUES (…),(…)`, **at most 1,000 rows per
statement** (T0-confirmed Q2). QuestDB reads the Bind parameter count as a signed int16, so the hard limit is
32,767 parameters. **(as built)** The live writer binds 8 parameters per row (`deleted` is the literal `false`; 4,095 rows
would fit), the updater 9 (3,640 rows). Within one statement a duplicate key keeps the **last** row, which is
why step 1's "first wins" is done client-side.
4. Stored-value rules are shared with SQL Server:
- NaN/±Inf, or any Empty with a non-Bad status, is stored Empty with `BadOutOfRange` or `BadNoData`.
- Comms-loss outage markers are stored as given.
- A tag name over 400 characters is a per-tag failure.
- **Ruling RC1 (T0 Q13):** a sample timestamped before 1970-01-01 cannot be stored, because QuestDB rejects
the whole INSERT. The writer drops such samples before the INSERT, logs one Warning per batch and counts them
(`DroppedPre1970`); the call still succeeds for the rest. A retry could
never succeed, and sending the sample would push the tag's whole group into the #538 quarantine.
5. Any SQL or transport failure makes the write return `false`, so the recorder retries and the #538 per-tag
quarantine still applies. It never throws.
The recorder is Primary-only (#537), so step 2 does not race another live writer. The recorder waits for WAL
visibility, bounded by `VisibilityTimeout` (plan R5), using the §2.3 wait.
Binding (T0-confirmed Q1/Q6/Q13):
- Every timestamp parameter is `NpgsqlDbType.TimestampTz` with `DateTimeKind.Utc`, after `Round` (**as built:** Unspecified is
taken as UTC, Local is converted, then ticks are truncated to a multiple of 10). `BindTimestamp(…, clampToEpoch: true)` is used
only for write binds of a designated timestamp; read bounds are never clamped (a pre-1970 bound is valid, and clamping it would
move the bound across the epoch row). `Timestamp` + `Utc`
throws in Npgsql. Npgsql truncates sub-µs ticks **toward 2000-01-01**, which rounds *up* before 2000. A raw
`DateTime.MinValue`/`MaxValue` is sent as ±infinity, and a `[Min, Max)` range then silently matches nothing. R6's
`BindTimestamp` (round; `MinValue` → `+10` ticks; `MaxValue` → `−9` ticks) fixes all three.
- The other parameter types:
| Column | `NpgsqlDbType` |
|---|---|
| `tag`/`str_value`/other strings | `Varchar` |
| `kind`/`data_type` | `Smallint` |
| `num_value` | `Double` |
| `int_value`/`status` | `Bigint` |
| `deleted` | `Boolean` |
| `severity` | `Integer` |
NULL is `DBNull.Value` with the column's type.
- Every QuestDB server error carries **SQLSTATE `00000`**. `QuestDbConnection`'s classification therefore treats a
`PostgresException` as a server-side rejection, judged by `MessageText`. Any other `NpgsqlException`, `IOException`,
`SocketException` or timeout is transport (transient).
### 2.3 HistoryUpdate (soft delete)
Every operation is an `INSERT`, and the dedup upsert performs the replace. QuestDB `UPDATE` and `DELETE` are never
issued. "Exists" means a row that is present **and not deleted**.
| Operation | Effect | Result |
|---|---|---|
| Insert | Write the row when none exists; **a tombstone counts as absent, so Insert over a tombstone returns `GoodEntryInserted`** (an explicit client insert may re-populate a deleted time; only the live writer's redelivery is barred, §2.2) | `GoodEntryInserted` / `BadEntryExists` |
| Replace | Overwrite the row when one exists | `GoodEntryReplaced` / `BadNoEntryExists` |
| Update | Write the row either way | `GoodEntryReplaced` or `GoodEntryInserted` |
| DeleteAtTime | Write a tombstone (`deleted = true`, value NULL) where a row exists | `Good` / `BadNoEntryExists` |
| DeleteRaw | Write a tombstone for every live row in the range | the same codes SQL Server returns for the same cases |
Further rules:
- The per-item rules come from `HistoryUpdateResults`, shared with SQL Server, so the codes are identical.
- Item order is preserved, with exactly one code per item.
- **Read-your-writes.** After its inserts, the updater polls `wal_tables()` until the table's applied transaction
reaches the sequencer transaction it produced, up to `VisibilityTimeout`. On a timeout the write is kept, the codes
are reported as normal, and a Warning is logged.
- **The visibility wait (`QuestDbWalVisibility`, T0-confirmed Q4)** is used by the writer and the updater (R5):
1. After the INSERT(s) return: `SELECT sequencerTxn FROM wal_tables() WHERE name = $1` (`$1` = the full table name,
`Varchar`) → `target`. Each autocommitted INSERT is exactly one sequencer transaction.
2. Poll `SELECT writerTxn, suspended FROM wal_tables() WHERE name = $1` about every 10 ms until `writerTxn >=
target`. Stop early, with a Warning, if `suspended` is true. Stop at `VisibilityTimeout`.
- Measured: median 0 ms, p95 6 ms, max 13 ms. Up to 7/20 inserts were not visible to an immediate re-read, so the
wait is required. A concurrent writer can only raise `target`. The key-poll fallback (S11) is not used.
- **(as built)** Existence checks for item times (Insert/Replace/Update/DeleteAtTime) are **not** `IN` lists: they use the T0 Q11 range
form, one query per `QuestDbTimeClusters` cluster, `SELECT ts FROM {p}sample WHERE tag = $1 AND ts >= $2 AND ts <= $3 AND NOT
deleted`, and the set test is client-side. The `IN ($2, …)` form this section originally proposed is not used.
- **Unknown tag (as built):** Insert and Update register the tag in `QuestDbTagCatalog`; Replace and the deletes do not (Replace ⇒
`BadNoEntryExists`, DeleteAtTime ⇒ `BadNoEntryExists`, DeleteRaw ⇒ `BadNoData`).
- **Tombstone shape (as built):** `kind = Empty (0)`, value columns NULL, `status = BadNoData`, `deleted = true`.
- **DeleteRaw (T0-confirmed), page size 1,000:**
1. `SELECT ts FROM {p}sample WHERE tag = $1 AND ts >= $2 AND ts < $3 AND NOT deleted ORDER BY ts LIMIT 1000`.
2. Write one 1,000-row tombstone INSERT.
3. Run the visibility wait.
4. Continue from `last ts + 10 ticks`, until a page comes back empty.
The wait between pages is required, or the next page re-reads rows just tombstoned. Measured: 2,500 rows, 3 pages,
103 ms.
- **Ruling RC1 (T0 Q13):** an Insert, Replace or Update item timestamped before 1970-01-01 returns
`BadOutOfRange`, because QuestDB cannot store it (nothing is sent; a null or `MinValue` timestamp is still
`BadInvalidTimestamp`, and **as built** a pre-1970 item that is also NaN reports `BadOutOfRange`, not `BadInvalidArgument`,
because the pre-1970 filter runs first). DeleteAtTime there returns `BadNoEntryExists`; DeleteRaw finds nothing. A DeleteRaw
*window* may start before 1970. Pre-1970 conformance seeds are skipped for QuestDB.
- **Concurrency (as built).** The updater serializes calls **per tag, in-process only** (64 striped async locks, held from the
existence read through the visibility wait), so two concurrent Insert/Replace/Delete batches on one tag cannot both act on a stale
read. It does **not** lock against the live writer or a second node. The existence checks read committed data, so a live recorder
sample at the *same* timestamp still in flight in the WAL can race the updater, and **two nodes writing the same new key
concurrently can produce last-write-wins** (QuestDB has no `UPDLOCK`; the Primary gate on the recorder makes it rare). Updates
normally target past timestamps, so this residual is accepted and documented.
### 2.4 Retention
When `RetentionDays` is set, the bootstrapper applies TTL to `hist_sample` and `hist_event`, and re-applies it when
the configured value changes. QuestDB then drops expired day partitions, tombstones included. When unset, data is
kept forever. `hist_tag` never expires.
T0-confirmed (Q5):
- **Create:** `… PARTITION BY DAY TTL n DAYS WAL …` (§2.1).
- **Change:** `ALTER TABLE {p}sample SET TTL n DAYS`. **Remove** (`RetentionDays` unset while the table has a TTL):
`ALTER TABLE … SET TTL 0 DAYS`.
- `n` cannot be a `$` parameter (`invalid syntax, should be <number> <unit> but was $1`). It is the validated integer,
formatted with `InvariantCulture`.
- TTL is not allowed on the `PARTITION BY YEAR` `hist_tag` table (`TTL value must be an integer multiple of the
partition size`).
- **TTL is enforced only on write commits (as built, N3).** QuestDB looks for expired partitions when a write commits to the table,
so a quiet table never drops anything; old partitions linger until the next write.
- **Reading the current TTL:** `SELECT ttlValue, ttlUnit FROM tables() WHERE table_name = $1`. QuestDB **normalises
the unit**: 7 days reads `1 WEEK`, 14 days `2 WEEK`, 30 days `30 DAY`, none `0 HOUR`. The bootstrapper compares in
hours (`HOUR`=1, `DAY`=24, `WEEK`=168; any other unit counts as different, RC3) and issues an `ALTER` only on a
difference. A large `RetentionDays` that QuestDB normalises to MONTH/YEAR therefore re-`ALTER`s on every boot (harmless).
`AutoCreateSchema=false` touches no TTL.
- `ALTER … SET TTL` is applied through the WAL, so `tables()` shows the old value for 25–100 ms. The bootstrapper does
not re-read to verify.
- Expiry is checked against the **wall clock**: the image default is `cairo.ttl.use.wall.clock=true`. A future-dated
sample does not expire current data (T0 Q5c).
- **Table listing:** `SELECT table_name FROM tables() WHERE starts_with(table_name, $1)`. Never `LIKE 'hist_%'`,
because `_` is a wildcard.
- **Concurrent bootstrap:** `CREATE TABLE IF NOT EXISTS` from 4 connections × 20 rounds produced no errors. The S11
defence stays: catch a `PostgresException` whose `MessageText` contains `already exists`, re-check `tables()`, and
rethrow if the table is absent. Never key on SQLSTATE (always `00000`).
## 3. Reads
Every query filters `NOT deleted`. Every value goes through `StoredSample`, so the quality clamp is the same as SQL
Server's.
- **Raw.** A range on `ts`, `ORDER BY ts [DESC]`, `LIMIT @top` (`maxValuesPerNode == 0` ⇒ unbounded), at most `top` samples, `ContinuationPoint: null`; the node manager pages by re-reading from the last timestamp (plan R8). The dedup key forbids same-instant ties. Bounds follow the existing data-source contract: forward `[start, end)`, reverse `(end, start]`
descending, and **(as built)** `start == end` reads the sample exactly at that (rounded) time. **(as built)** The data source reports
`TimestampResolutionTicks = 10`, and the node manager steps off the last drained timestamp by that amount (forward `T + 10`, reverse
`T − 10`): a 1-tick step is truncated back to `T`, so a one-value-per-page forward read would loop forever. The rule is *not* "trim
everything ≤ the last emitted", because the same paging serves the Gateway backend, where same-timestamp ties are legal.
- **T0-confirmed form (review fix):** `SELECT ts, kind, num_value, int_value, str_value, status FROM {p}sample WHERE
tag = $1 AND ts >= $2 AND ts < $3 AND NOT deleted ORDER BY ts [DESC] LIMIT $4`.
- `$4` is a **bound** parameter, `NpgsqlDbType.Integer` (Int32) or `Bigint` (Int64); both work on 10.0.1, ASC and
DESC. Use `Integer`, since `top` is an `int`.
- `int.MaxValue` (the R8 "unbounded" value) and `long.MaxValue` both return every row. `LIMIT 0` returns no rows,
but R8 never sends 0.
- The inlined-literal fallback is not needed.
- **AtTime (T0-confirmed Q9).** Find the previous-or-equal neighbour and the next neighbour of each requested time
with `LIMIT 1` seeks, `UNION ALL`-ed, **at most 50 request times (100 branches) per statement**. The exact QuestDB
10.0.1 limit is **126 `UNION ALL` branches per statement (63 request times)**: 126 work, and 127 fail with `SQL
model is too complex to evaluate`. So 50 times leaves a 26-branch margin. Per time `k`, with `$1` = tag and `$j` = the
rounded time:
```sql
SELECT * FROM (SELECT k AS req, 0 AS dir, ts, kind, num_value, int_value, str_value, status FROM {p}sample
WHERE tag = $1 AND ts <= $j AND NOT deleted ORDER BY ts DESC LIMIT 1)
UNION ALL
SELECT * FROM (SELECT k AS req, 1 AS dir, ts, kind, num_value, int_value, str_value, status FROM {p}sample
WHERE tag = $1 AND ts > $j AND NOT deleted ORDER BY ts ASC LIMIT 1)
```
`k` and `dir` are generated integer literals. Measured: 102 times in about 16–20 ms over 100k rows per tag, with no
mismatches. `ASOF JOIN` was faster (1 ms), but it yields only the previous neighbour, so it was not chosen.
`AtTimeEvaluator` then decides the result. **(as built, M3)** Each time is evaluated at its *rounded* value but the result is stamped
with the time the client *requested* (UTC-normalised, not truncated), as on SQL Server, so answers match requests. Stored data is µs-only,
so a sub-µs request adjacent to a sample is an exact hit.
- **Events.** Filtered by source and `ts` range `[start, end)`, ordered `(ts, event_id)` ascending (reversed windows are swapped, as built
today). **(as built)** With a bound, `maxEvents + 1` rows are fetched; the extra row is dropped and becomes the continuation point.
- **Processed.** **One query per node** (T0-confirmed for every family; the S11 two-statement fallback is not needed).
- Bucket index (T0-confirmed): `(cast(ts AS LONG) * 10 - $1) / $2`.
- `cast(ts AS LONG)` is µs since 1970.
- `$1` = the rounded window start in **ticks since the Unix epoch** (`start.Ticks − DateTime.UnixEpoch.Ticks`).
- `$2` = the interval in ticks. Both are `NpgsqlDbType.Bigint`.
- The result is exact because both sides are µs-aligned (R6), and it cannot overflow before year 9999.
- **(as built)** The bucket timestamp returned for a Processed bucket is the *rounded* window start plus `i × interval`, so with a
sub-µs request start it differs from SQL Server's label by the sub-µs remainder; values and membership are identical.
- The query returns the full `BucketRow` statistics. Usable, good, bad and uncertain counts come from
`sum(CASE …)` over the status column.
- It does **not** use `SAMPLE BY`, whose alignment rules differ from ours. Computing the index ourselves keeps the
bucket edges identical to SQL Server: start inclusive, end exclusive, a partial final bucket, interval 0, and the
100k cap.
- The time of the min or max value (ActualTime family) comes from a `row_number()` window function in the same single
statement: `ORDER BY num_value ASC|DESC, ts ASC`, with ties going to the earliest sample (T0-confirmed; the form is
below). No second statement is used. `AggregateEvaluator` then produces the values.
- **Confirmed forms (T0 Q8).** They mirror `SqlAggregateQueries`'s three families and column order. They were checked
against hand-computed seeds and a 60k-row randomized cross-check across 4 partitions: 617 buckets, 0 mismatches.
- Classified subquery `s`:
```sql
SELECT (cast(ts AS LONG) * 10 - $1) / $2 AS b, ts, kind, num_value, int_value, str_value,
CASE WHEN status < 2147483648 AND kind <> 0 THEN 1 ELSE 0 END AS u,
CASE WHEN status < 1073741824 AND kind <> 0 THEN 1 ELSE 0 END AS g
FROM {p}sample WHERE tag = $3 AND ts >= $4 AND ts < $5 AND NOT deleted
```
- Counts (ordinals 1–5 after `b`): `count() AS total_count, sum(u) AS usable_count, sum(g) AS good_count, count() -
sum(u) AS bad_count, sum(CASE WHEN u = 1 AND kind IN (4, 5) THEN 1 ELSE 0 END) AS non_numeric_count`.
- **Simple:** `SELECT b, <counts>, avg(CASE WHEN u = 1 THEN num_value END) AS avg_value, min(CASE WHEN u = 1 THEN
num_value END) AS min_value, max(…) AS max_value, sum(…) AS sum_value FROM (<s>) GROUP BY b ORDER BY b`.
- **StartEnd:**
```sql
WITH s AS (<s>), st AS (SELECT b, <counts> FROM s GROUP BY b),
f AS (SELECT b, kind, num_value, int_value, str_value, ts,
row_number() OVER (PARTITION BY b ORDER BY ts) AS rf,
row_number() OVER (PARTITION BY b ORDER BY ts DESC) AS rl FROM s WHERE u = 1)
SELECT st.*, x.kind, x.num_value, x.int_value, x.str_value, x.ts, y.kind, y.num_value, y.int_value, y.str_value, y.ts
FROM st LEFT JOIN (SELECT * FROM f WHERE rf = 1) x ON x.b = st.b
LEFT JOIN (SELECT * FROM f WHERE rl = 1) y ON y.b = st.b ORDER BY st.b
```
- **ActualTime:** the same `s`/`st`, then `r AS (SELECT b, num_value, ts, row_number() OVER (PARTITION BY b ORDER
BY num_value ASC|DESC, ts ASC) AS rn FROM s WHERE u = 1 AND num_value IS NOT NULL) SELECT st.*, r.num_value, r.ts
FROM st LEFT JOIN (SELECT * FROM r WHERE rn = 1) r ON r.b = st.b ORDER BY st.b`.
- A bucket with no usable sample returns NULL times but **`kind = 0`** (BYTE has no NULL). The mapper keys absence on
the NULL `ts`.
- `first()/last()` and `arg_min/arg_max` also returned correct values, but no `ORDER BY` fixes their tie order. The
window forms state the earliest-timestamp tie-break explicitly and are the ones to use.
- **Errors.** A failed read fails only that node, under the same data-source contract as SQL Server.
## 4. Configuration (`ServerHistorian:QuestDb`)
| Key | Default | Validation |
|---|---|---|
| `ConnectionString` | — | Required when `Backend=QuestDb` and a consumer is enabled. An Npgsql string. **T0-confirmed:** `Host=questdb;Port=8812;Username=admin;Password=quest;Database=qdb;ServerCompatibilityMode=NoTypeLoading` (the docker-dev overlay from the host: `Host=localhost;Port=18812;…`). `ServerCompatibilityMode=NoTypeLoading` is kept: it skips the `pg_type` load, and every confirmed form was run with it. The string also works without it. `No Reset On Close` is **not** needed, because pooled open → close → reopen works. |
| `TablePrefix` | `hist_` | `^[A-Za-z0-9_]+$`. Only checked when a consumer is enabled. |
| `RetentionDays` | unset | ≥ 1 when set |
| `CommandTimeout` | `00:00:30` | > 0 |
| `AutoCreateSchema` | `true` | — |
| `VisibilityTimeout` | `00:00:05` | ≥ 0. 0 disables the wait. |
The Gateway keys (`Endpoint`, `ApiKey`, `UseTls`) are not required for QuestDb, the same as for SqlServer.
**(as built) Pool size.** QuestDB's default limit is **64** PG-wire connections; Npgsql's default `Maximum Pool Size` is **100**, and the
read, write, update and alarm paths share one pool per node. Operators should set `Maximum Pool Size` in the connection string below the
server limit, per node (a redundant pair shares one QuestDB). The validator checks the prefix only when a consumer is enabled, but the
connection helper also refuses a malformed prefix (`ArgumentException`) when the first seam is built.
## 5. Testing
- **T1 refactor gate.** The SQL Server integration suite (118) and unit suite pass unchanged before and after the
extraction. Each moved piece of logic is falsified once against its existing test.
- **Conformance suite** (`tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Conformance`, a class library of abstract
xUnit tests). Each backend's integration project subclasses it and supplies a fixture and an `IConformanceSeeder`,
which writes rows directly in that backend's schema.
- It covers raw forward, reverse and paging; AtTime, including the outage, after-last and future rules; all 13
aggregates with the existing hand-computed literals; the quality contract; insert-if-absent writes and
redelivery; the HistoryUpdate result codes; and over-long tag names.
- Each legitimate backend difference is a named, documented override, never a silent skip. Microsecond rounding is
the only expected one.
- **QuestDB-only tests:**
- a tombstone hides the row and a redelivery does not resurrect it;
- the read-your-writes wait;
- TTL is applied, verified through `tables()` metadata;
- concurrent bootstrap from two instances;
- intra-batch microsecond collisions;
- the `hist_tag` upsert;
- transient-failure behaviour, writing against a stopped port;
- **(as built)** pre-1970 handling (writer drop, `BadOutOfRange`, read bounds), the `long.MinValue` read repair, Insert over a tombstone,
per-tag updater serialization, and a forward cap-1 paging regression against a real QuestDB.
- Conformance differences are named overrides: pre-1970 seeds are skipped on QuestDB, and AtTime timestamp assertions compare `Round(x)` to
the requested time (identity on SQL Server). The QuestDB fixture prunes idle pool connections after 1 s because of the 64-connection limit.
- **Fixture.** `HISTORIAN_QUESTDB_TEST_CONNECTION`; when it is absent, the tests skip. Each test class uses a unique
throwaway `TablePrefix` (`t_<guid>_`) and drops its tables on dispose.
- **Host.** Registry and validator tests for `Backend=QuestDb`, beside the SqlServer ones.
## 6. Environment
- QuestDB is an **optional overlay file**, `docker-dev/docker-compose.questdb.yml`, so the default rig is unchanged.
It defines the `questdb` service:
- image `questdb/questdb:10.0.1`, pinned by T0 and pushed 2026-08-24;
- container `otopcua-dev-questdb-1`;
- started with `docker compose -f docker-dev/docker-compose.yml -f docker-dev/docker-compose.questdb.yml up -d
--no-deps questdb`.
- It exposes PG wire on host port **18812** (→ 8812) and the web console/REST on host port **19000** (→ 9000), so it
does not collide with AdminUI `:9000`/`:9200`.
- It uses the named volume `otopcua-questdb-data`.
- The central nodes gain `ServerHistorian__QuestDb__ConnectionString` (`OTOPCUA_HISTORIAN_QUESTDB`), defaulting to the
`questdb` service.
- No new NuGet packages.
## 7. Live gate (T9, run by the controller)
On docker-dev, with `Backend=QuestDb`:
1. Only the Primary records, with real kinds and status, one row per value.
2. Raw and Processed HistoryRead return identical results through the raw and UNS NodeIds, and match a direct
QuestDB query.
3. HistoryUpdate covers insert, duplicate, replace, replace-missing and delete-at-time, each confirmed by a read and
by a direct query. After a delete, an outbox redelivery does not bring the row back. A WriteOperate-only key and an
anonymous session are both denied.
4. On failover, recording continues at 1×.
5. Modbus device loss stores a `BadCommunicationError` Empty sample, then the `BadNoCommunication` outage marker,
then Good on recovery.
6. TTL is visible in table metadata when `RetentionDays` is set.
Evidence goes in `docs/plans/2026-10-10-questdb-historian-live-gate.md`, and the rig is restored afterwards.
## 8. Work order
| Wave | Task | Model |
|---|---|---|
| W0 | **T0 spike.** Pin the image, add the compose service, and prove every QuestDB assumption here: Npgsql parameterised multi-row inserts, `DEDUP UPSERT`, `wal_tables()` visibility, TTL syntax, microsecond timestamps over Npgsql, the min/max-time query form, AtTime neighbour lookup, concurrent `CREATE TABLE IF NOT EXISTS`. **Amend §2–§4 with the findings before W1.** | controller + sonnet |
| W1 | T1: extract `Sql.Common`, leaving SQL Server's behaviour unchanged | opus |
| W2 | T2 conformance suite over SQL Server ‖ T3 QuestDB skeleton (options, connection, bootstrapper, fixture) | sonnet ‖ sonnet |
| W3 | T4 writes, provisioning and alarms ‖ T5 reads and aggregates ‖ T6 updater with soft delete | sonnet ‖ opus ‖ sonnet |
| W4 | T7 provider, registry and validator, plus the conformance suite run against QuestDB | sonnet |
| W5 | T8 docs (Historian.md, Configuration.md, CLAUDE.md, docker-dev README), then a Fable whole-branch review, one fix wave, and T9 live gate | sonnet / fable / opus / controller |
docs/plans/2026-10-10-questdb-historian-live-gate.md
+205
View File
@@ -0,0 +1,205 @@
# QuestDB historian — live gate (T9), 2026-10-11
**Verdict: PASSED.** All six legs of spec §7 passed on the docker-dev rig. The rig ran `feat/questdb-historian` at
`2ebe0a42`, with the central pair on image `0701f6a833ea`, rebuilt for this gate. The QuestDB backend pointed at the
overlay `questdb` service (`questdb/questdb:10.0.1`, `docker-dev/docker-compose.questdb.yml`), with default table
prefix `hist_`.
The gate also produced two findings, neither in the QuestDB backend. Both are described under "Gate findings" below:
- A crash failover (SIGKILL of the recording Primary) leaves a recording gap of about 30 s. This is the #537
Primary-gated capture waiting for Akka downing; it is backend-independent.
- One `libgssapi_krb5.so.2` load error is printed per central boot. It is benign.
## Setup
- `docker-dev/docker-compose.yml` gained `ServerHistorian__QuestDb__RetentionDays`
(`OTOPCUA_HISTORIAN_QUESTDB_RETENTION_DAYS`, default empty = no TTL) on both centrals, beside the existing
`ServerHistorian__QuestDb__ConnectionString` (`OTOPCUA_HISTORIAN_QUESTDB`).
- Central-1 and central-2 were started with:
```
OTOPCUA_HISTORIAN_ENABLED=true OTOPCUA_HISTORIAN_BACKEND=QuestDb OTOPCUA_CONTHIST_ENABLED=true \
docker compose -f docker-dev/docker-compose.yml -f docker-dev/docker-compose.questdb.yml up -d --no-deps central-1 central-2
```
With `RetentionDays` empty, both nodes booted cleanly; the empty value binds as "no TTL". Each logged
`QuestDB historian connection pool: Maximum Pool Size = 24 (default cap).` (the final-review I-1 fix).
- Cluster `MAIN` fixtures on `10.100.0.35`: opc-plc (`opcplc/plc/FastUInt1`, UInt32, about 1 Hz) and pymodbus
`standard` (`pymodbus/plc/HR5`).
- Gate configuration, written to the config DB and deployed headlessly (`deploy.sh`, deployment `0b4f5208`):
- New Calculation String tag `Calc/eng/FastLabel` = `"v" + FastUInt1` (script `MAIN-script-fastlabel`).
- `FastUInt1`, `FastLabel` and `HR5` were marked `isHistorized`.
- A UNS reference `a/l/e1/FastUInt1` was added.
- Tags were provisioned at deploy (`dispatched=3, requested=3, ensured=3, skipped=0, failed=0` on both nodes). `hist_tag`:
| name | data_type |
|---|---|
| `Calc/eng/FastLabel` | 9 (String) |
| `opcplc/plc/FastUInt1` | 5 (Int64) |
| `pymodbus/plc/HR5` | 4 |
- Tables at first boot, from `tables()`:
- `hist_sample`, `hist_event`: WAL, `PARTITION BY DAY`, `ttlValue 0`.
- `hist_tag`: WAL, `PARTITION BY YEAR`.
- Two data-plane API keys were minted for the gate, each with a `Cluster`-scoped grant. The ACL flag stayed off, which
is the rig default.
- `gate-historyupdate`: ReadOnly | HistoryUpdate (4239)
- `gate-writeop`: ReadOnly | WriteOperate (159)
Direct queries used QuestDB REST: `curl -G localhost:19000/exec --data-urlencode "query=…"`.
## 1. Recording: Primary-only, typed, real status, one row per value
- **Rows carry real kinds and statuses:**
- `kind=2` Int64 `int_value 278` (with `num_value 278.0`), status `0`.
- `kind=4` String `v278`, status `0`.
- At startup, `kind=0` Empty with `2150760448` (`0x80320000` BadWaitingForInitialData) and `2150694912`
(`0x80310000` BadNoCommunication). These match the SQL Server gate.
- **Primary-only:**
- Central-2 was the recording Primary, and its outbox `hlog.log.0` grew.
- Central-1 logged "this node is not the recording Primary … suppressing value capture". Its `hlog.log.0` kept its
2026-10-09 19:13 mtime and was not written.
- **One row per value**, from 07:43:20 onwards:
- 57 FastUInt1 rows over 57 distinct timestamps; 57 FastLabel rows over 57.
- All 57 pair one-to-one: same `ts`, and `str_value = 'v' + int_value`.
- **Outbox carry-over across a backend switch.** Two rows carry source time `2026-10-09T19:13:40.129127Z` (`296` /
`v296`) with `server_ts 07:42:27`, before the deploy. They were un-drained entries left in central-2's outbox by the
2026-10-09 SQL Server gate, and they drained into QuestDB at boot. An outbox entry outlives the backend it was
captured under, by design: the outbox is backend-neutral.
## 2. HistoryRead through the raw and UNS NodeIds
All reads went to central-1 (`:4840`).
- **Raw read**, 07:44:00–07:44:10, on `ns=2;s=opcplc/plc/FastUInt1` and on `ns=3;s=a/l/e1/FastUInt1`:
- Both return the identical 10 rows, `271`…`280`, all Good.
- They equal the direct query `select ts, int_value, status from hist_sample where tag=… and not deleted` exactly,
at microsecond precision (for example `07:44:00.952990Z`).
- **Average at 10 s**, 07:44:00–07:44:30, through both NodeIds:
- Both return `275.5 / 285.5 / 295.5`, all Good.
- The direct `avg(int_value)` per 10 s bucket gives `275.5 / 285.5 / 295.5`, over 10, 10 and 10 samples.
## 3. HistoryUpdate
These calls went to central-1 on `ns=2;s=opcplc/plc/FastUInt1`, with `-U apikey`. Each was checked by `historyread`
and by a direct query.
| Call | Result | Confirmed |
|---|---|---|
| insert 06:00:00 = 42 (`gate-historyupdate`) | `GoodEntryInserted` | row `42`, status 0, `deleted=false` |
| repeat insert = 43 | `BadEntryExists` | row still `42` |
| replace 06:00:00 = 99 | `GoodEntryReplaced` | row `99` |
| replace 06:00:30 (no entry) | `BadNoEntryExists` | — |
| update 06:01:00 = 11 (absent) | `GoodEntryInserted` | row `11` |
| delete-at-time 06:00:00 | `Good` | tombstone row: `int_value NULL`, status `0x809B0000`, `deleted=true` |
| insert over the tombstone, 06:00:00 = 55 | `GoodEntryInserted` (ledger ruling: an insert over a tombstone is an insert) | read returns `55` |
| delete-at-time 06:00:00 again | `Good` | read returns only `06:01:00 = 11` |
| insert with `gate-writeop` (WriteOperate only) | Node result `BadUserAccessDenied` | no row |
| insert, anonymous | Node result `BadUserAccessDenied` | no row |
| insert at 1969-12-31 (RC1) | `BadOutOfRange` | no row |
No QuestDB `UPDATE` or `DELETE` was issued: every change above is a new row under `DEDUP UPSERT KEYS(ts, tag)`.
### A delete is not resurrected by an outbox redelivery
This was proven with a real replay, not a simulation:
1. **07:47:26–07:47:35:** `docker stop` QuestDB. The Primary (central-1) kept capturing; its writes failed, so the
entries stayed pending in its outbox.
2. **07:47:35:** the outbox was snapshotted while QuestDB was down (commit point `commit.1303.0`).
3. **07:47:35:** QuestDB was restarted. The 16-value backlog (`476`…`491`) drained as one batch (`server_ts
07:47:41.620267`), with no gap. This is a second result: QuestDB loss is ridden out by the outbox.
4. **07:50:45–46:** `delete-at-time` on two backlog rows, `07:47:28.950591` (`479`) and `07:47:31.949216` (`482`).
Both became tombstones (`deleted=true`, `0x809B0000`).
5. **07:50:46:** central-1 was stopped. Its outbox directory (on the `otopcua-localdb-central-1` volume) was replaced
with the pre-delivery snapshot.
6. **07:51:15:** central-1 was started. It re-drained the snapshot's entries, and the outbox commit point advanced
`1303 → 1328`.
After the replay:
- Both tombstones are unchanged: same `server_ts 07:50:45.669146 / 07:50:46.074579`, still `deleted=true`.
- Every other row in the window kept its original `server_ts 07:47:41.620267`, so nothing was re-written.
- 24 rows over 12 keys (2 tags × 12): no duplicates.
- `historyread` 07:47:27–07:47:33 returns `478, 480, 481, 483`; `479` and `482` stay deleted.
A first attempt used `ALTER TABLE hist_sample SUSPEND WAL` plus a SIGKILL of the Primary. It could not produce a
redelivery. The writer deliberately treats a suspended table as committed and acknowledges it (R5: rows committed to
the WAL become visible later, and are not re-sent). It logged "QuestDB table hist_sample is suspended (writerTxn 327
< sequencerTxn 337); not waiting" and acked. That attempt exposed the crash-failover finding below.
## 4. Failover
- **07:50:46:** central-1 was stopped (step 5 above). Central-2 logged "now the recording Primary" at 07:50:46.
- **07:51:57:** `docker stop` central-2, then the Primary. Central-1 logged "this node is now the recording Primary …
resuming value capture" at 07:51:57, as central-2 began leaving.
- **07:52:13:** central-2 was restarted and rejoined as standby ("not the recording Primary"). Its outbox
`hlog.log.0` stayed frozen at 57856 bytes (mtime 07:51:58). Central-1's grew from 65536 to 84480.
- **Rate across both failovers was 1×.** Per-30 s FastUInt1 counts from 07:50:00 to 07:54:30:
30, 29, 30, 30, 30, 30, 29, 29, 30.
- There was no gap at the boundary: `743`…`754` are contiguous across 07:51:57.
- There was no doubling: 267 rows over 267 distinct timestamps.
## 5. Device loss: driver Bad sample and comms-loss outage marker
This leg ran on Modbus (`HR5`). `otopcua-pymodbus-standard` was stopped at 07:54:40 and restarted at 07:56:51.
| ts | kind | value | status | server_ts |
|---|---|---|---|---|
| 07:51:18.715101 | 2 | 5 | `0` | 07:51:18 |
| 07:54:51.960244 | 0 (Empty) | NULL | `0x80050000` BadCommunicationError (the driver's own Bad publish) | 07:56:13 |
| 07:56:48.652185 | 0 (Empty) | NULL | `0x80310000` BadNoCommunication (the #527 watchdog outage marker) | 07:56:48 |
| 07:56:58.688150 | 2 | 5 | `0` (recovery re-publish) | 07:56:58 |
Reads over the outage:
- Raw `historyread` 07:50–07:58 shows Good, Bad, Bad, Good.
- Average at 120 s, 07:52–07:58, gives `(null) Bad` / `(null) Bad` / `5 Uncertain`. The outage is visible and is not
bridged.
The Bad sample's `server_ts` is 82 s after its source time. Its source time is the driver's failed-read stamp; it
reached the recorder later. Its position in the series is correct either way, because ordering and reads key on
source time.
## 6. TTL visible in table metadata
Both centrals were recreated with `OTOPCUA_HISTORIAN_QUESTDB_RETENTION_DAYS=30`. At 07:57:41 the bootstrapper logged
`Applied TTL 30 day(s) to QuestDB table hist_sample.` and the same for `hist_event`. The TTL was applied to the
already existing tables.
`tables()` before and after:
| table | before | after |
|---|---|---|
| `hist_event` | 0 HOUR | 30 DAY |
| `hist_sample` | 0 HOUR | 30 DAY |
| `hist_tag` | 0 HOUR | 0 HOUR (no TTL on the tag catalog, by design) |
`SHOW CREATE TABLE hist_sample`:
`… timestamp(ts) PARTITION BY DAY TTL 30 DAYS DEDUP UPSERT KEYS(ts,tag);`
Recording resumed after the restart: 57 FastUInt1 rows and 56 FastLabel rows in the first minute.
## Gate findings
1. **A crash failover leaves a recording gap of about 30 s. This is backend-independent and predates this work.**
- **What happened:** after `docker kill` (SIGKILL) of the recording Primary at 07:45:58, the next FastUInt1 row
is at 07:46:29. Central-1 became the recording Primary only at 07:46:30, when the killed node rejoined as a new
incarnation.
- **Why:** #537 gates capture to the Primary. A crashed Primary stays unreachable but not yet downed for the Akka
failure-detector window plus `auto-down-unreachable-after = 15s`, so no node captures during that time.
- **Contrast:** a clean `docker stop` hands over at once (leg 4). The SQL Server gate only exercised the clean
path.
- **Status:** proposed follow-up issue, not fixed here.
2. **`Cannot load library libgssapi_krb5.so.2` is printed once per central boot.** It does not appear on the site
nodes, which have neither a ConfigDb nor a QuestDB client. It is most likely Npgsql's default GSS-encryption probe,
which falls back cleanly. Everything worked. If it is confirmed as Npgsql, `GSS Encryption Mode=Disable` in the
connection string silences it.
## Restore
- The rig was restored afterwards:
- central-1 and central-2 were recreated with the compose defaults (historian off, `Backend=Gateway`).
- The gate tag, script, UNS reference, API keys and ACL rows were removed.
- The `HR5` and `FastUInt1` TagConfigs were reverted.
- The configuration was redeployed.
- The overlay `questdb` container was stopped and removed; the `otopcua-questdb-data` volume was kept.
- The opc-plc and pymodbus fixtures were stopped (normally down).
- The compose keys stay: `Backend` defaults to `Gateway`, and `RetentionDays` defaults to empty.
docs/plans/2026-10-10-questdb-historian-plan.md
+1080
View File
File diff suppressed because it is too large Load Diff
docs/plans/2026-10-10-questdb-historian-plan.md.tasks.json
+96
View File
File diff suppressed because one or more lines are too long
docs/plans/2026-10-10-questdb-spike.md
+451
View File
@@ -0,0 +1,451 @@
# QuestDB historian backend — T0 feasibility spike
**Status:** run 2026-10-11 against a live container. Gitea #666. Spec: `docs/plans/2026-10-10-questdb-historian-design.md`
(§2–§4 amended in place from this doc; every confirmed form there is marked **T0-confirmed**).
## Fixture
| Item | Value |
|---|---|
| Image | **`questdb/questdb:10.0.1`** — Docker Hub `tag_last_pushed` **2026-08-24T14:00:08Z** (48 days before 2026-10-10) |
| Why that tag | Newest exact `x.y.z` tag ≥ 14 days older than 2026-10-10. Newer tags are `nightly*` / `patch-*` only. Floor for TTL is **8.2.3**; 10.0.1 is well above it. |
| Build | `SELECT build()` → `Build Information: QuestDB 10.0.1, JDK 25.0.2, Commit Hash 7a391566ac827e0d8c269b91f2415c16b0a32a84` |
| Overlay | `docker-dev/docker-compose.questdb.yml` — container `otopcua-dev-questdb-1`, ports `18812:8812` (PG wire) and `19000:9000` (console/REST), named volume `otopcua-questdb-data` (explicit `name:` so it is not project-prefixed) |
| Up | `docker compose -f docker-dev/docker-compose.yml -f docker-dev/docker-compose.questdb.yml up -d --no-deps questdb` (a `--dry-run` showed only the volume + `otopcua-dev-questdb-1` being created; nothing else in the `otopcua-dev` project is touched) |
| Healthcheck | the image ships `/usr/bin/curl` (`command -v curl` → `/usr/bin/curl`; no `wget`), so `curl -fs http://localhost:9003/status` is kept. Container reached `healthy`. |
| Harness | throwaway net10.0 console app, Npgsql **10.0.3** (the repo's central pin, `Directory.Packages.props` line 97), in the session scratchpad `questdb-spike/` with its own `Directory.Build.props`, `Directory.Packages.props` and `nuget.config`. Nothing was added to the repo. |
### Connection string (confirmed)
```
Host=localhost;Port=18812;Username=admin;Password=quest;Database=qdb;ServerCompatibilityMode=NoTypeLoading
```
Inside the docker network the same string uses `Host=questdb;Port=8812`.
## Q1 — keywords, parameter types, CLR types
```
[OK] open+SELECT 1 with 'Host=localhost;Port=18812;Username=admin;Password=quest;Database=qdb'
[OK] open+SELECT 1 with '...;ServerCompatibilityMode=NoTypeLoading'
[OK] open->close->reopen x5 on one data source (no 'No Reset On Close')
[OK] reopen after a parameterised INSERT (prepared-ish state)
[FAIL] ts Utc + Timestamp: ArgumentException: Cannot write DateTime with Kind=UTC to PostgreSQL type 'timestamp without time zone', consider using 'timestamp with time zone'. ...
[OK] ts Utc + TimestampTz
[OK] ts Utc + inferred
[OK] ts Unspecified + Timestamp
[FAIL] ts Local + TimestampTz: ArgumentException: Cannot write DateTime with Kind=Local to PostgreSQL type 'timestamp with time zone', only UTC is supported. ...
[OK] kind bound as Integer (int4)
[OK] tag bound as Text
[OK] NaN double insert
[OK] +Inf double insert
cols: ts:timestamp without time zone/DateTime, tag:character varying/String, kind:smallint/Int16, num_value:double precision/Double,
int_value:bigint/Int64, str_value:character varying/String, status:bigint/Int64, deleted:boolean/Boolean, server_ts:timestamp without time zone/DateTime
2026-01-02T03:04:05.1234560(Unspecified) | ts Utc + TimestampTz | 3<Int16> | 1.5 | 7<Int64> | s | 2147483648<Int64> | False | 2026-10-11T04:56:27.1103710(Unspecified)
...
2026-01-02T03:04:11.1234560(Unspecified) | nan | 0<Int16> | NULL | NULL | NULL | NULL | False | NULL
2026-01-02T03:04:12.1234560(Unspecified) | inf | 0<Int16> | NULL | NULL | NULL | NULL | False | NULL
integer arithmetic with $n:
cols: ts::LONG - $1:bigint/Int64 [OK] cast(ts AS LONG) - $1 (Bigint)
cols: (ts::LONG * 10 - $1) / $2:bigint/Int64 [OK] (Bigint) [OK] (Integer)
```
Sentinel NULLs (extra check):
```
cols: ts, l:bigint/Int64, i:integer/Int32, b:smallint/Int16, z:boolean/Boolean, l = NULL:boolean
2026-03-01T00:00:00 | NULL | NULL | 0 | False | True <- l = long.MinValue, i = int.MinValue written
2026-03-01T00:00:01 | -9223372036854775807 | -2147483647 | 0 | False | False
2026-03-01T00:00:02 | NULL | NULL | 0 | False | True <- BYTE/BOOLEAN bound as DBNull
```
**Decisions**
- Both connection strings work. **Keep `ServerCompatibilityMode=NoTypeLoading`**: it skips the `pg_type` load, and every
form in this doc was run with it on. **`No Reset On Close` is not needed**: open → close → reopen on one pooled
`NpgsqlDataSource` works.
- **Timestamps are bound as `NpgsqlDbType.TimestampTz` with `DateTimeKind.Utc`.** `Timestamp` + `Utc` throws in Npgsql
before reaching the server. `Unspecified` + `Timestamp` also works, but one rule is enough.
- **Timestamps read back as `DateTime` with `Kind=Unspecified`**, wall clock = UTC. The reader must
`DateTime.SpecifyKind(r.GetDateTime(i), DateTimeKind.Utc)`.
- CLR types returned: `BYTE` → `smallint` / **`Int16`** (`GetInt16`); `SYMBOL` → `character varying` / `String`;
`VARCHAR` → `String`; `LONG` → `Int64`; `INT` → `Int32`; `BOOLEAN` → `Boolean`; `TIMESTAMP` → `DateTime`;
`DOUBLE` → `Double`. `count()`/`sum(int)` → `Int64`.
- Parameter types: `BYTE` ← `Smallint` (`Integer` also accepted); `SYMBOL`/`VARCHAR` ← `Varchar` (`Text` also accepted);
`LONG` ← `Bigint`; `BOOLEAN` ← `Boolean`; `DOUBLE` ← `Double`; a NULL is `DBNull.Value` with the column's
`NpgsqlDbType`.
- `$n` in integer arithmetic: **bind as `NpgsqlDbType.Bigint`** (Integer also worked for small values; Bigint is the
rule because tick offsets exceed int32).
- **DOUBLE NaN and ±Inf read back as NULL.** **`LONG` `long.MinValue` and `INT` `int.MinValue` are QuestDB's NULL
sentinels** and read back NULL. **`BYTE` and `BOOLEAN` have no NULL**: unset or DBNull reads `0` / `false`. So
`deleted` defaults to `false` and `kind` of a never-written field is `0` (Empty).
- **Every server error carries `SqlState = 00000`.** Classification cannot use SQLSTATE: a `PostgresException` is a
server-side rejection (decide by `MessageText`), any other `NpgsqlException`/`IOException`/`SocketException`/timeout
is transport.
## Q2 — multi-row parameterised INSERT and batch size
```
[OK] 1000 rows x 9 params = 9000 params 88 ms
[OK] 2000 rows x 9 params = 18000 params 167 ms
[OK] 3640 rows x 9 params = 32760 params 342 ms
[FAIL] 3641 rows x 9 params = 32769 params: PostgresException: 00000: invalid parameter count [parameterCount=-32767, offset=225670
[FAIL] 5000 rows x 9 params = 45000 params: PostgresException: 00000: invalid parameter count [parameterCount=-20536, ...
[FAIL] 7282 rows x 9 params = 65538 params: NpgsqlException: A statement cannot have more than 65535 parameters
20 x 1000-row statements: fails=0, 122 ms total
count()=27251 | count_distinct(ts)=27251 expected rows = 27251
```
**Decision:** QuestDB reads the Bind parameter count as a **signed int16**, so the real limit is **32,767 parameters per
statement**, not Npgsql's 65,535. **Batch size = 1,000 rows per `INSERT … VALUES (…),(…)`** (9,000 parameters for
`hist_sample`), which is reliable and far under the limit. **DeleteRaw page size = 1,000 rows** (see the DDL section).
## Q3 — `DEDUP UPSERT KEYS`
```
before: 2026-03-01T00:00:00 | a | 1 | False 2026-03-01T00:00:00 | b | 2 | False
after re-insert of (t0,'a') as tombstone:
2026-03-01T00:00:00 | a | NULL | True 2026-03-01T00:00:00 | b | 2 | False
NOT deleted: 2026-03-01T00:00:00 | b | 2
same key twice in one statement (10 then 20): 2026-03-01T00:00:01 | c | 20
hist_event with event_id VARCHAR as a dedup key:
[OK] CREATE hist_event-shape with DEDUP UPSERT KEYS(ts, event_id) on VARCHAR
[OK] insert e1 'first', e2, then e1 'second'
2026-03-01T00:00:00 | e1 | src1 | 500 | second | NULL
2026-03-01T00:00:00 | e2 | src1 | 500 | other | NULL
```
**Decision:** re-inserting a key replaces the whole row; a tombstone written at `(ts, tag)` hides the row from `NOT
deleted`. **Within one statement the LAST duplicate wins**, so the writer's "first value wins" rule must be applied
client-side before the INSERT (spec §2.2 step 1 already does this). **`event_id VARCHAR` works as a dedup key**, so the
S4 fallback (no dedup on `hist_event`) is **not needed**.
## Q4 — `wal_tables()` and the visibility wait
```
cols: name:varchar, suspended:boolean, writerTxn:bigint, bufferedTxnSize:bigint, sequencerTxn:bigint,
errorTag:varchar, errorMessage:varchar, memoryPressure:integer
run 0: seqTxn before=0 after=1 visibleImmediately=False waitMs=13 visibleAfterWait=True
run 1: seqTxn before=1 after=2 visibleImmediately=False waitMs=0 visibleAfterWait=True
...
run 16: seqTxn before=16 after=17 visibleImmediately=False waitMs=6 visibleAfterWait=True
lag ms: min=0 median=0 p95=6 max=13; visible-without-wait 17/20
```
(An earlier run: visible without waiting 13/20, max 6 ms.) Every autocommitted INSERT is exactly one sequencer
transaction.
**Decision (the form T3's `QuestDbWalVisibility` implements):**
1. After the INSERT(s) return, on the same or any connection:
`SELECT sequencerTxn FROM wal_tables() WHERE name = $1` (`$1` = the table name, `Varchar`) → `target`.
2. Poll `SELECT writerTxn, suspended FROM wal_tables() WHERE name = $1` every ~10 ms until `writerTxn >= target`, or
`suspended` is true, or `VisibilityTimeout` elapses.
A concurrent writer can only make `target` larger, which lengthens the wait slightly and never shortens it. The wait is
needed: up to 7/20 inserts were not yet visible on the next query. The S11 key-poll fallback is **not needed**.
## Q5 — TTL
```
[FAIL] CREATE ... PARTITION BY DAY WAL DEDUP UPSERT KEYS(ts, tag) TTL 7 DAYS (TTL last): unexpected token [TTL]
[OK] CREATE ... PARTITION BY DAY TTL 7 DAYS WAL DEDUP UPSERT KEYS(ts, tag) (TTL before WAL)
spike_q5 | DAY | True | True | ttlValue=1 | ttlUnit=WEEK
[FAIL] ALTER TABLE spike_q5 SET TTL $1 DAYS (parameter): invalid syntax, should be <number> <unit> but was $1
[FAIL] TTL 7 DAYS on PARTITION BY YEAR table: TTL value must be an integer multiple of the partition size (its time interval)
Q5b ALTER ... SET TTL visibility in tables()
SET TTL 30 DAYS : seqTxn 6->7; applied after 100 ms; tables() now ttlValue=30 ttlUnit=DAY
SET TTL 14 DAYS : seqTxn 7->8; applied after 94 ms; tables() now ttlValue=2 ttlUnit=WEEK
SET TTL 1 DAY : seqTxn 8->9; applied after 59 ms; tables() now ttlValue=1 ttlUnit=DAY
SET TTL 0 DAYS : seqTxn 9->10; applied after 25 ms; tables() now ttlValue=0 ttlUnit=HOUR
SET TTL 3 DAYS : seqTxn 10->11; applied after 60 ms; tables() now ttlValue=3 ttlUnit=DAY
after inserting now-20d and now into a TTL 7 DAYS table:
new | 2026-10-11T05:03:03
Q5c TTL reference clock (future-dated row):
after inserting a now+30d row, then one more ordinary commit:
d-3 | 2026-10-08 now | 2026-10-11 now+1h | 2026-10-11 d+30 | 2026-11-10
```
Image `conf/server.conf`: `#cairo.ttl.use.wall.clock=true` — "when true (default), TTL enforcement uses wall clock
time to prevent accidental data loss when future timestamps are inserted".
**Decision:**
- Create syntax: **`… TIMESTAMP(ts) PARTITION BY DAY TTL n DAYS WAL DEDUP UPSERT KEYS(…)`** — TTL goes **between
`PARTITION BY` and `WAL`**. Alter: **`ALTER TABLE t SET TTL n DAYS`**; remove: **`ALTER TABLE t SET TTL 0 DAYS`**.
`n` cannot be a parameter; it is the validated integer `RetentionDays`, formatted with `InvariantCulture`.
- Metadata: `SELECT ttlValue, ttlUnit FROM tables() WHERE table_name = $1`. **QuestDB normalises the unit**: 7 days
shows `1 WEEK`, 14 days `2 WEEK`, 30 days `30 DAY`, none `0 HOUR`. The bootstrapper compares in hours
(`HOUR`=1, `DAY`=24, `WEEK`=168; any other unit counts as different) and issues `ALTER` only when they differ.
- `ALTER … SET TTL` is a WAL transaction: `tables()` shows the old value until it is applied (25–100 ms). Do not
re-read to verify without the Q4 wait.
- Partitions older than TTL are dropped on commit. With the default wall-clock mode a future-dated sample does **not**
expire current data.
- TTL cannot go on the `PARTITION BY YEAR` `hist_tag` table; it is never needed there.
## Q6 — timestamp precision
```
raw sent 2026-03-01T12:00:00.1234567(Utc) -> read 2026-03-01T12:00:00.1234560 micros=1772366400123456 exact=False
raw9 sent 2026-03-01T12:00:01.1234569(Utc) -> read 2026-03-01T12:00:01.1234560 micros=1772366401123456 exact=False
pre sent 2026-03-01T12:01:00.1234560(Utc) -> read 2026-03-01T12:01:00.1234560 micros=1772366460123456 exact=True
```
**Decision:** Npgsql **truncates** sub-µs ticks (…1234567 → …123456; not rounded). That is truncation **toward zero
relative to 2000-01-01**, so before 2000 it rounds *up* (Q13). A pre-truncated (`ticks - ticks % 10`) value
round-trips exactly. This confirms R6: always `Round` first.
## Q7 — concurrent `CREATE TABLE IF NOT EXISTS`
```
Q7 concurrent CREATE TABLE IF NOT EXISTS (4 connections x 20 rounds)
errors: none
plain CREATE on existing: SqlState=00000 MessageText='table already exists'
```
(An earlier run with 2 connections × 20 rounds also had no errors.) **Decision:** `IF NOT EXISTS` was race-free here.
The S11 catch rule stays as defence: catch a `PostgresException` whose `MessageText` contains **`already exists`**,
then re-check with `SELECT count() FROM tables() WHERE table_name = $1`, and rethrow when the table is not there. Never
key on SQLSTATE (it is always `00000`).
## Q8 — bucket statistics
Seed (3 × 10 s buckets from `start = t0 + 0.5 s`): bucket 0 holds Good 5, Uncertain 2, Bad 9, a usable String, Good 2
(a min tie with the Uncertain 2), a tombstone of 100, and Good 7 at the last µs of the bucket. Bucket 1 holds Good 1 at
the edge, an Empty `BadNoCommunication` and an Empty with Good status. Bucket 2 holds Bad 3 only. Another tag sits in
range.
```
Simple family (GROUP BY alias):
cols: b, total_count, usable_count, good_count, bad_count, non_numeric_count, avg_value, min_value, max_value, sum_value
0 | 6 | 5 | 4 | 1 | 1 | 4 | 2 | 7 | 16
1 | 3 | 1 | 1 | 2 | 0 | 1 | 1 | 1 | 1
2 | 1 | 0 | 0 | 1 | 0 | NULL | NULL | NULL | NULL
[OK] simple GROUP BY <expression> (no subquery) [OK] simple implicit group-by
StartEnd form A (first()/last()):
0 | 6 | 5 | 4 | 1 | 1 | 1 | 5 | NULL | NULL | 00:00:00.5000000 | 1 | 7 | NULL | NULL | 00:00:10.4999990
1 | 3 | 1 | 1 | 2 | 0 | 1 | 1 | NULL | NULL | 00:00:10.5000000 | 1 | 1 | NULL | NULL | 00:00:10.5000000
2 | 1 | 0 | 0 | 1 | 0 | 0 | NULL | NULL | NULL | NULL | 0 | NULL | NULL | NULL | NULL
StartEnd form B (row_number window): identical rows [OK] filter-in-subquery [OK] extra predicate in ON
ActualTime (min): 0 ... | 2 | 00:00:01.5000000 1 ... | 1 | 00:00:10.5000000 2 ... | NULL | NULL
ActualTime (max): 0 ... | 7 | 00:00:10.4999990 1 ... | 1 | 00:00:10.5000000 2 ... | NULL | NULL
[OK] arg_min(ts, num_value) exists? 0 | 00:00:01.5 | 00:00:10.499999 1 | 00:00:10.5 | 00:00:10.5
randomized cross-check (60k rows over 3 days (4 partitions), 2 tags, 7 min buckets, random statuses/kinds/tombstones):
buckets=617 rowsInWindow=28423 mismatches=0; first()/last() 23 ms, window 21 ms, actual-time min+max 26 ms;
arg_min/arg_max (one stmt) 11 ms, tie-break mismatches vs earliest-ts rule = 0
```
Every hand-computed value matches: bucket 0 averages (5+2+2+7)/4 = 4, the tombstone is invisible, the sample one µs
before the edge stays in bucket 0, the edge sample is in bucket 1, and both Empty rows count as bad whatever their
status. The randomized cross-check compares counts, first/last usable times, sums and min/max actual times against a
client-side computation. It used a µs-aligned start that is not on a second boundary, and integer-rounded values to
force ties.
**Decision (the confirmed forms, with the SQL Server family split kept):**
- Classified subquery `s`. Status is stored as the **unsigned** uint in a LONG, so usable = `status < 2147483648` and
good = `status < 1073741824`, both also needing `kind <> 0`:
```sql
SELECT (cast(ts AS LONG) * 10 - $1) / $2 AS b, ts, kind, num_value, int_value, str_value,
CASE WHEN status < 2147483648 AND kind <> 0 THEN 1 ELSE 0 END AS u,
CASE WHEN status < 1073741824 AND kind <> 0 THEN 1 ELSE 0 END AS g
FROM {p}sample WHERE tag = $3 AND ts >= $4 AND ts < $5 AND NOT deleted
```
`$1` = window start in **ticks since the Unix epoch** (`start.Ticks - DateTime.UnixEpoch.Ticks`, `Bigint`); `$2` =
interval in ticks (`Bigint`); `$4`/`$5` = rounded window bounds. `cast(ts AS LONG)` is µs since 1970, so `*10` is
ticks. That is exact because both sides are already µs-aligned (R6). At year 9999 `µs*10` ≈ 2.5e18 < 9.2e18, so it
cannot overflow.
- Common counts: `count() AS total_count, sum(u) AS usable_count, sum(g) AS good_count, count() - sum(u) AS bad_count,
sum(CASE WHEN u = 1 AND kind IN (4, 5) THEN 1 ELSE 0 END) AS non_numeric_count`.
- **Simple:** `SELECT b, <counts>, avg(CASE WHEN u = 1 THEN num_value END) AS avg_value, min(…), max(…), sum(…) FROM (<s>)
GROUP BY b ORDER BY b`.
- **StartEnd:** the window form, which is the same shape as SQL Server:
`WITH s AS (<s>), st AS (SELECT b, <counts> FROM s GROUP BY b), f AS (SELECT b, kind, num_value, int_value,
str_value, ts, row_number() OVER (PARTITION BY b ORDER BY ts) AS rf, row_number() OVER (PARTITION BY b ORDER BY ts
DESC) AS rl FROM s WHERE u = 1) SELECT st.*, x.kind, x.num_value, x.int_value, x.str_value, x.ts, y.kind, …, y.ts
FROM st LEFT JOIN (SELECT * FROM f WHERE rf = 1) x ON x.b = st.b LEFT JOIN (SELECT * FROM f WHERE rl = 1) y ON
y.b = st.b ORDER BY st.b`. A bucket without a usable sample has NULL `ts` but **kind `0`** (BYTE has no NULL), so the
evaluator must key "no first/last" on the NULL time.
- **ActualTime:** `… r AS (SELECT b, num_value, ts, row_number() OVER (PARTITION BY b ORDER BY num_value ASC|DESC, ts
ASC) AS rn FROM s WHERE u = 1 AND num_value IS NOT NULL) SELECT st.*, r.num_value, r.ts FROM st LEFT JOIN (SELECT *
FROM r WHERE rn = 1) r ON r.b = st.b ORDER BY st.b`.
- **One query per node holds for every family.** The S11 two-statement fallback is not needed. `first()/last()` and
`arg_min/arg_max` also gave correct answers, but their tie order is not specified by an `ORDER BY`. The window form
states the earliest-timestamp tie-break explicitly, so it is the one to use.
- `SAMPLE BY` is not used.
## Q9 — AtTime neighbour lookup
200k rows (2 tags × 100k, ~1.5 s apart, 5 % tombstones). 102 request times, including one before the first sample,
one exact hit and one after the last sample.
```
form A SQL (one time): SELECT * FROM (SELECT 0 AS req, 0 AS dir, ts, kind, num_value, int_value, str_value, status FROM spike_q9
WHERE tag = $1 AND ts <= $2 AND NOT deleted ORDER BY ts DESC LIMIT 1) UNION ALL SELECT * FROM (SELECT 0 AS req, 1 AS dir, ts, ...
WHERE tag = $1 AND ts > $2 AND NOT deleted ORDER BY ts ASC LIMIT 1)
form A chunk= 1: 61 ms for 102 times, rows=202, mismatches=0
form A chunk= 10: 20 ms ... chunk= 25: 18 ms chunk= 50: 19 ms chunk= 60: 18 ms chunk= 63: 16 ms (all mismatches=0)
form A chunk= 64: FAIL 'SQL model is too complex to evaluate' (128 UNION ALL branches)
form A chunk=102: FAIL 'SQL model is too complex to evaluate' (204 UNION ALL branches)
form B: 1 ms for 102 times (prev only), rows=102, mismatches=0 [ASOF JOIN of a long_sequence(1) UNION ALL request set]
```
**Decision: form A, chunked `LIMIT 1` seeks, at most 50 request times (100 `UNION ALL` branches) per statement.**
The exact limit is **126 `UNION ALL` branches (63 request times) per statement**. A re-measure with single-seek
branches (review fix, below) showed 126 work and 127 fail. The form A runs above agree: 63 times (126 branches) work,
and 64 times (128 branches) fail. The literal `req` index and `dir` (0 = previous-or-equal, 1 = next) columns are
generated integers, not user input; `$1` is the tag and `$2..` the rounded times. It returns both neighbours in one
shape and costs ~0.2 ms per time. ASOF JOIN is faster, but it only gives the previous neighbour, so a second
seek form would still be needed for the next one. That does not justify two code paths at this scale.
## Q10 — long strings and SYMBOL capacity
```
[OK] 400-char tag (SYMBOL) + 400-char str_value (VARCHAR)
[OK] 5000-char str_value
30,000 distinct SYMBOL values inserted in 723 ms
count()=30002 | count_distinct(tag)=30002
400-char round trip: tag equal=True str equal=True lengths=400,400 (includes non-ASCII αβγ…)
tag | SYMBOL | symbolCached=True | symbolCapacity=16384
point lookup by tag among 30k symbols: count=1 0 ms
```
**Decision:** VARCHAR and SYMBOL take 400 characters (and VARCHAR takes 5,000). SYMBOL grows past its capacity hint
(16,384) without error, so 30k tags is fine. No `CAPACITY` clause is needed.
## Q11 — existing-timestamp lookup for a batch
```
[OK] range form: WHERE tag=$1 AND ts >= $2 AND ts <= $3 (tombstones included) -> 10 rows, incl. 00:00:08 deleted=True
[OK] IN-list form: ts IN ($2..$11) -> 4 rows (00, 06, 12, 18)
[OK] tombstone visible to range lookup -> 00:00:08 | True
```
**Decision:** both work with parameters. **Live writer (§2.2 step 2): the range form**
`SELECT ts FROM {p}sample WHERE tag = $1 AND ts >= $2 AND ts <= $3` (no `NOT deleted`, so tombstones count as present),
then a client-side set test. **Updater (sparse item times): the IN-list form** `SELECT ts, deleted FROM {p}sample WHERE
tag = $1 AND ts IN ($2, …)`, chunked at 1,000 times.
## Q12 — test cleanup and prefix listing
```
LIKE 't12_%' (note: _ is a wildcard) -> t12_sample, t12xsample <- WRONG: matches t12xsample
LIKE $1 with escaped _ ('t12\_%') -> t12_sample
starts_with(table_name, $1) -> t12_sample
left(table_name, 4) = $1 -> t12_sample
[OK] DROP TABLE IF EXISTS t12_sample (exists) [OK] DROP TABLE IF EXISTS t12_sample (already gone)
```
**Decision:** cleanup is `DROP TABLE IF EXISTS {name}`. List with **`SELECT table_name FROM tables() WHERE
starts_with(table_name, $1)`**. Never use `LIKE 'prefix%'`, because `_` in `hist_` is a wildcard.
## Q13 — binding extreme DateTimes
Seeds: 1970-01-01, 1990-05-05T00:00:00.123456, 2026-03-01, 2200-01-01. "Round" is R6's rule: `MinValue` → `+10 ticks`,
then `ticks - ticks % 10`.
```
DateTime.MinValue raw -> server sees µs -9222425352054775808 (Npgsql sends -infinity)
DateTime.MinValue Round -> 0001-01-01T00:00:00.0000010 (µs -62135596799999999)
DateTime.MaxValue raw -> server sees µs -9222425352054775809 (Npgsql sends +infinity)
DateTime.MaxValue Round -> 9999-12-31T23:59:59.9999990 (µs 253402300799999999)
0001-01-01 raw -> µs -9222425352054775808 (== MinValue -> -infinity)
1601-01-01 raw/Round -> 1601-01-01T00:00:00 (µs -11644473600000000)
9999-12-31T23:59:59.9999999 raw -> µs -9222425352054775809 (== MaxValue -> +infinity)
1969-12-31T23:59:59.9999995 raw -> 1970-01-01T00:00:00.0000000 (µs 0) Round -> 1969-12-31T23:59:59.9999990 (µs -1)
1990-05-05T00:00:00.1234567 raw -> 1990-05-05T00:00:00.1234570 Round -> ...1234560
1990-05-05T00:00:00.1234565 raw -> 1990-05-05T00:00:00.1234570 Round -> ...1234560
2026-03-01T00:00:00.0000005 raw -> 2026-03-01T00:00:00.0000000 Round -> same
range queries WHERE ts >= $1 AND ts < $2:
Min..Max raw count=0 <- silently empty
Min..Max Round count=4
0001..9999.9999999 Round count=4
1601..t0 Round count=2
1990.1234565..1990.1234575 raw count=0 <- seed .123456 missed (bound arrived as .123457)
1990.1234565..1990.1234575 Round count=1
inserting extreme designated timestamps:
[FAIL] insert ts=1969-12-31T23:59:59: PostgresException: 00000: designated timestamp before 1970-01-01 is not allowed
[FAIL] insert ts=1601-01-01: ... [FAIL] insert ts=Min+10 ticks: ...
[OK] insert ts=9999-12-31 Round -> 9999-12-31T23:59:59.9999990
```
**Decision:** R6's `BindTimestamp` is **confirmed necessary and sufficient**.
- A raw `DateTime.MinValue`/`MaxValue` (and anything equal to them) becomes ±infinity, and QuestDB stores that as a
near-`long.MinValue` value. A `[Min, Max)` range then matches **nothing**, with no error.
- A raw pre-2000 sub-µs value rounds **up**, which shifts a bound across a stored sample.
- With Round, every bound behaves.
- **New constraint:** a designated timestamp **before 1970-01-01** cannot be inserted. Reads with pre-1970 bounds work.
See the spec §2.2 ruling candidate.
## DDL and DeleteRaw paging (extra)
```
[OK] CREATE TABLE IF NOT EXISTS spike_hist_sample (...) TIMESTAMP(ts) PARTITION BY DAY TTL 30 DAYS WAL DEDUP UPSERT KEYS(ts, tag)
[OK] CREATE TABLE IF NOT EXISTS spike_hist_event (...) TIMESTAMP(ts) PARTITION BY DAY TTL 30 DAYS WAL DEDUP UPSERT KEYS(ts, event_id)
[OK] CREATE TABLE IF NOT EXISTS spike_hist_tag (...) TIMESTAMP(ts) PARTITION BY YEAR WAL DEDUP UPSERT KEYS(ts, name)
[OK] re-run (idempotent) x3
spike_hist_event | DAY | True | True | 30 | DAY
spike_hist_sample | DAY | True | True | 30 | DAY
spike_hist_tag | YEAR | True | True | 0 | HOUR
hist_tag upsert-by-name at ts=1970-01-01 (tagA twice, tagB once):
1970-01-01T00:00:00 | tagA | 4 | degF | second
1970-01-01T00:00:00 | tagB | 1 | NULL | NULL
DeleteRaw paging (page = 1000 rows): 2500 live rows + 1 outside range
pages=3 tombstones=2500 in 103 ms
deleted=False 1 deleted=True 2500
```
DeleteRaw loop, as confirmed: `SELECT ts FROM {p}sample WHERE tag = $1 AND ts >= $2 AND ts < $3 AND NOT deleted ORDER BY
ts LIMIT 1000` → one 1,000-row tombstone INSERT → the Q4 wait → continue from `last ts + 10 ticks`. The wait matters:
without it the next page can re-read rows it just tombstoned.
## Review fixes (2026-10-11)
### Raw `LIMIT $n` with a bound parameter
`dotnet run -- limit` against 10.0.1, with 10 rows seeded in `[t0, t0+10 s)`. The SQL was `SELECT ts, num_value FROM
spike_lim WHERE tag = $1 AND ts >= $2 AND ts < $3 AND NOT deleted ORDER BY ts [DESC] LIMIT $4`:
```
rows=3 first=0 last=2 [OK] ASC LIMIT $4 (Integer 3)
rows=3 first=9 last=7 [OK] DESC LIMIT $4 (Integer 3)
rows=3 first=0 last=2 [OK] ASC LIMIT $4 (Bigint 3)
rows=3 first=9 last=7 [OK] DESC LIMIT $4 (Bigint 3)
rows=10 first=0 last=9 [OK] ASC LIMIT $4 (Integer int.MaxValue)
rows=10 first=9 last=0 [OK] DESC LIMIT $4 (Integer int.MaxValue)
rows=10 first=0 last=9 [OK] ASC LIMIT $4 (Bigint long.MaxValue)
rows=10 first=9 last=0 [OK] DESC LIMIT $4 (Bigint long.MaxValue)
rows=0 first=NULL last=NULL [OK] ASC LIMIT $4 (Integer 0)
rows=0 first=NULL last=NULL [OK] DESC LIMIT $4 (Integer 0)
```
**Decision:** a bound `LIMIT $n` works, with `NpgsqlDbType.Integer` (Int32) or `Bigint` (Int64). Use `Integer`, since
`top` is an `int`. R8's unbounded value `int.MaxValue` returns every row. `LIMIT 0` returns nothing, but R8 never sends
it. The inlined-literal fallback is not needed.
### Exact `UNION ALL` branch limit
`dotnet run -- union`. Each branch is `SELECT * FROM (SELECT k AS req, ts FROM spike_un WHERE tag = $1 AND ts <= $2 AND
NOT deleted ORDER BY ts DESC LIMIT 1)`:
```
[OK] 100 branches [OK] 120 branches [OK] 125 branches [OK] 126 branches
[FAIL] 127 branches: PostgresException: 00000: SQL model is too complex to evaluate
[FAIL] 128 branches / 129 / 130: same
```
**Decision:** the limit is **126 branches, which is 63 AtTime request times** (two branches per time). The chosen chunk
of **50 times (100 branches)** is unchanged.
## Rulings candidates raised by the spike
1. **Pre-1970 timestamps cannot be stored** (Q13). QuestDB rejects a designated timestamp before 1970-01-01, and the
whole multi-row INSERT fails. Proposed:
- the live writer drops such samples and logs one Warning per batch. Retrying cannot succeed, and sending the
sample would fail the tag's whole group into #538 quarantine;
- HistoryUpdate Insert/Replace/Update items before 1970 return `BadOutOfRange`;
- DeleteAtTime/DeleteRaw find nothing there and return `BadNoEntryExists`;
- reads are unaffected.
Conformance tests that seed pre-1970 data are skipped for QuestDB.
2. **`long.MinValue` is QuestDB's LONG NULL** (sentinel check). An Int64 sample of exactly `long.MinValue` would read
back `int_value` NULL, and `FromColumns` would then serve **0**. Proposed: the QuestDB read path repairs it. When
`kind = Int64`, `int_value IS NULL` and `num_value = -9.223372036854775808E18` (exact in a double), serve
`long.MinValue`. That is a local fix-up before the shared `FromColumns`, with no write-side change.
3. **TTL metadata is unit-normalised and applied asynchronously** (Q5). The bootstrapper compares in hours and does not
re-read straight after an `ALTER`.
src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/Historian/IHistorianDataSource.cs
+15
View File
@@ -107,4 +107,19 @@ public interface IHistorianDataSource : IDisposable
/// </summary>
/// <returns>The current health snapshot for the historian backend.</returns>
HistorianHealthSnapshot GetHealthSnapshot();
/// <summary>
/// The backend's storage timestamp resolution, in <see cref="DateTime"/> ticks (100 ns): the smallest step
/// between two distinct stored timestamps. <c>1</c> (the default) means full tick precision; QuestDB stores
/// microseconds and reports <c>10</c>.
/// </summary>
/// <remarks>
/// The server's HistoryRead-Raw paging steps OFF a drained timestamp by this amount (forward
/// <c>T + resolution</c>, reverse <c>T − resolution</c>). Stepping by less than the resolution is not
/// progress on a backend that truncates its inputs: <c>T + 1 tick</c> is truncated back to <c>T</c>, the
/// same sample comes back, and a one-value-per-page forward read loops forever. Stepping by the resolution is
/// still lossless, because no stored timestamp can lie strictly between <c>T</c> and <c>T ± resolution</c>.
/// Same-timestamp ties are unaffected (they are paged within the timestamp, never stepped over).
/// </remarks>
long TimestampResolutionTicks => 1;
}
src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbAggregateQueries.cs
+122
View File
@@ -0,0 +1,122 @@
using System.Globalization;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb;
/// <summary>
/// Push-down QuestDB SQL for processed (aggregate) reads (spec §3 "Processed", T0 Q8-confirmed forms), one query
/// per <see cref="AggregateFamily"/>, mirroring the SQL Server backend's <c>SqlAggregateQueries</c> family split and
/// column order. Every query takes five positional parameters:
/// <list type="number">
/// <item><c>$1</c> — the (rounded) window start in 100 ns ticks since the Unix epoch
/// (<see cref="BucketPlan.StartUnixTicks"/>, <c>Bigint</c>);</item>
/// <item><c>$2</c> — the bucket width in ticks (<see cref="BucketPlan.IntervalTicks"/>, <c>Bigint</c>);</item>
/// <item><c>$3</c> — the tag (<c>Varchar</c>);</item>
/// <item><c>$4</c>/<c>$5</c> — the forward window <c>[start, end)</c> (<c>TimestampTz</c> via
/// <see cref="QuestDbConnection.BindTimestamp"/>).</item>
/// </list>
/// Each returns one row per <b>non-empty</b> bucket; the caller fills the empty ones.
/// </summary>
/// <remarks>
/// <para>
/// <b>Bucketing.</b> <c>b = (cast(ts AS LONG) * 10 − $1) / $2</c>. <c>cast(ts AS LONG)</c> is µs since 1970, so
/// <c>× 10</c> is ticks since 1970; both it and <c>$1</c> are µs-aligned (R6), so the integer division is
/// start-inclusive / end-exclusive to the tick, exactly like SQL Server's. Every row is at or after the start,
/// so the dividend is never negative (no truncation-toward-zero surprise), and nothing overflows before 9999.
/// <c>SAMPLE BY</c> is deliberately not used: its alignment rules differ from ours.
/// </para>
/// <para>
/// <b>Sample classes (S6).</b> The status is stored as the <b>unsigned</b> 32-bit code in a <c>LONG</c>, so
/// "not Bad" is <c>status &lt; 0x80000000</c> (not T-SQL's sign test). <i>usable</i> = not Bad AND
/// <c>kind &lt;&gt; 0</c> (an Empty sample is an outage marker and counts as Bad whatever its code);
/// <i>good</i> = usable AND <c>status &lt; 0x40000000</c>; <i>bad</i> = not usable; <i>non-numeric</i> = usable
/// String/DateTime. The literals are generated from <see cref="StatusBits"/> and <see cref="HistorianValueKind"/>.
/// </para>
/// <para>Common result columns (ordinals 0–5): <c>b, total_count, usable_count, good_count, bad_count,
/// non_numeric_count</c>. Family columns follow from <see cref="FirstFamilyColumn"/>.</para>
/// </remarks>
internal static class QuestDbAggregateQueries
{
/// <summary>Ordinal of the first family-specific column.</summary>
public const int FirstFamilyColumn = 6;
/// <summary><c>status &lt; SeverityBadBit</c>, as a decimal LONG literal.</summary>
internal static readonly string NotBadLimit = StatusBits.SeverityBadBit.ToString(CultureInfo.InvariantCulture);
/// <summary><c>status &lt; SeverityUncertainBit</c>, as a decimal LONG literal.</summary>
internal static readonly string GoodLimit = StatusBits.SeverityUncertainBit.ToString(CultureInfo.InvariantCulture);
/// <summary>The <c>kind</c> values of String and DateTime: usable but not numeric.</summary>
internal static readonly string NonNumericKinds =
((byte)HistorianValueKind.String).ToString(CultureInfo.InvariantCulture) + ", " +
((byte)HistorianValueKind.DateTime).ToString(CultureInfo.InvariantCulture);
private static readonly string EmptyKind = ((byte)HistorianValueKind.Empty).ToString(CultureInfo.InvariantCulture);
/// <summary>
/// Average / Minimum / Maximum / Total / Count / Range / PercentGood / PercentBad.
/// Family columns: <c>6 avg_value, 7 min_value, 8 max_value, 9 sum_value</c> (DOUBLE, NULL when no usable sample).
/// </summary>
/// <param name="sample">Prefixed sample table name.</param>
/// <returns>The SQL text.</returns>
public static string Simple(string sample) =>
"SELECT b, " + CommonCounts() + ", " +
"avg(CASE WHEN u = 1 THEN num_value END) AS avg_value, " +
"min(CASE WHEN u = 1 THEN num_value END) AS min_value, " +
"max(CASE WHEN u = 1 THEN num_value END) AS max_value, " +
"sum(CASE WHEN u = 1 THEN num_value END) AS sum_value " +
"FROM (" + Classified(sample) + ") GROUP BY b ORDER BY b";
/// <summary>
/// Start / End / Delta: the first and last usable sample of each bucket by <c>ts</c>.
/// Family columns: <c>6..10</c> first sample (<c>kind, num_value, int_value, str_value, ts</c>), <c>11..15</c> last
/// sample (same order). A bucket without a usable sample has NULL <c>ts</c> but <c>kind = 0</c> (BYTE has no NULL),
/// so the reader keys absence on the NULL time.
/// </summary>
/// <param name="sample">Prefixed sample table name.</param>
/// <returns>The SQL text.</returns>
public static string StartEnd(string sample) =>
"WITH s AS (" + Classified(sample) + "), " +
"st AS (SELECT b, " + CommonCounts() + " FROM s GROUP BY b), " +
"f AS (SELECT b, kind, num_value, int_value, str_value, ts, " +
"row_number() OVER (PARTITION BY b ORDER BY ts ASC) AS rf, " +
"row_number() OVER (PARTITION BY b ORDER BY ts DESC) AS rl FROM s WHERE u = 1) " +
"SELECT st.*, " +
"x.kind, x.num_value, x.int_value, x.str_value, x.ts, " +
"y.kind, y.num_value, y.int_value, y.str_value, y.ts " +
"FROM st LEFT JOIN (SELECT * FROM f WHERE rf = 1) x ON x.b = st.b " +
"LEFT JOIN (SELECT * FROM f WHERE rl = 1) y ON y.b = st.b ORDER BY st.b";
/// <summary>
/// MinimumActualTime / MaximumActualTime: the usable sample with the smallest (largest) <c>num_value</c>, ties
/// broken by the earliest <c>ts</c>. Family columns: <c>6 num_value, 7 ts</c> (NULL when the bucket has no usable
/// numeric sample).
/// </summary>
/// <param name="sample">Prefixed sample table name.</param>
/// <param name="maximum">True for MaximumActualTime.</param>
/// <returns>The SQL text.</returns>
public static string ActualTime(string sample, bool maximum) =>
"WITH s AS (" + Classified(sample) + "), " +
"st AS (SELECT b, " + CommonCounts() + " FROM s GROUP BY b), " +
"r AS (SELECT b, num_value, ts, " +
"row_number() OVER (PARTITION BY b ORDER BY num_value " + (maximum ? "DESC" : "ASC") + ", ts ASC) AS rn " +
"FROM s WHERE u = 1 AND num_value IS NOT NULL) " +
"SELECT st.*, r.num_value, r.ts " +
"FROM st LEFT JOIN (SELECT * FROM r WHERE rn = 1) r ON r.b = st.b ORDER BY st.b";
/// <summary>The bucket-index expression (exposed for tests).</summary>
internal const string BucketExpression = "(cast(ts AS LONG) * 10 - $1) / $2";
/// <summary>The classified subquery <c>s</c>: the window's live samples with bucket index and the usable/good flags.</summary>
private static string Classified(string sample) =>
"SELECT " + BucketExpression + " AS b, ts, kind, num_value, int_value, str_value, " +
$"CASE WHEN status < {NotBadLimit} AND kind <> {EmptyKind} THEN 1 ELSE 0 END AS u, " +
$"CASE WHEN status < {GoodLimit} AND kind <> {EmptyKind} THEN 1 ELSE 0 END AS g " +
$"FROM {sample} WHERE tag = $3 AND ts >= $4 AND ts < $5 AND NOT deleted";
/// <summary>The five common count columns (after <c>b</c>).</summary>
private static string CommonCounts() =>
"count() AS total_count, sum(u) AS usable_count, sum(g) AS good_count, count() - sum(u) AS bad_count, " +
$"sum(CASE WHEN u = 1 AND kind IN ({NonNumericKinds}) THEN 1 ELSE 0 END) AS non_numeric_count";
}
src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbAlarmHistorianWriter.cs
+130
View File
@@ -0,0 +1,130 @@
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Logging.Abstractions;
using Npgsql;
using NpgsqlTypes;
using ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian;
using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb;
/// <summary>
/// <see cref="IAlarmHistorianWriter"/> over the QuestDB event table. Each event carries the deterministic
/// <see cref="AlarmEventIds.ComputeEventId"/> and is inserted through the table's <c>DEDUP UPSERT KEYS(ts, event_id)</c>
/// (T0 Q3 confirmed VARCHAR works as a key), so the store-and-forward drain's at-least-once redelivery, and both
/// pair nodes writing the same event, converge on one row (the redelivered row is identical except
/// <c>received_utc</c>). Outcomes: Ack on insert; RetryPlease on transient / unclassified errors and on
/// cancellation; PermanentFail only for data the server will always reject (a time before 1970, which is not a
/// storable designated timestamp, or a server message naming an inconvertible / out-of-range value). Never throws.
/// </summary>
internal sealed class QuestDbAlarmHistorianWriter : IAlarmHistorianWriter
{
private static readonly string[] PermanentMessageFragments =
["designated timestamp", "inconvertible", "out of range", "too long", "cannot be null"];
private readonly QuestDbConnection _conn;
private readonly QuestDbSchemaBootstrapper _bootstrap;
private readonly QuestDbWalVisibility _wal;
private readonly ILogger _log;
private readonly TimeProvider _clock;
/// <summary>Creates the writer.</summary>
/// <param name="conn">Connection helper.</param>
/// <param name="bootstrap">Schema bootstrapper.</param>
/// <param name="wal">WAL visibility wait (once per batch, so a following read sees the events).</param>
/// <param name="log">Optional logger.</param>
/// <param name="clock">Clock for <c>received_utc</c>.</param>
public QuestDbAlarmHistorianWriter(QuestDbConnection conn, QuestDbSchemaBootstrapper bootstrap, QuestDbWalVisibility wal,
ILogger? log = null, TimeProvider? clock = null)
{
_conn = conn;
_bootstrap = bootstrap;
_wal = wal;
_log = log ?? NullLogger.Instance;
_clock = clock ?? TimeProvider.System;
}
/// <inheritdoc/>
public async Task<IReadOnlyList<HistorianWriteOutcome>> WriteBatchAsync(IReadOnlyList<AlarmHistorianEvent> batch, CancellationToken cancellationToken)
{
ArgumentNullException.ThrowIfNull(batch);
if (batch.Count == 0) return [];
var outcomes = new HistorianWriteOutcome[batch.Count];
try
{
await _bootstrap.EnsureAsync(cancellationToken).ConfigureAwait(false);
}
catch (Exception ex)
{
_log.LogWarning("QuestDB alarm historian bootstrap failed ({Exception}); will retry.", ex.GetType().Name);
Array.Fill(outcomes, HistorianWriteOutcome.RetryPlease);
return outcomes;
}
var wrote = false;
for (var i = 0; i < batch.Count; i++)
{
if (cancellationToken.IsCancellationRequested) { outcomes[i] = HistorianWriteOutcome.RetryPlease; continue; }
outcomes[i] = await WriteOneAsync(batch[i], cancellationToken).ConfigureAwait(false);
wrote |= outcomes[i] == HistorianWriteOutcome.Ack;
}
if (wrote)
{
try { await _wal.WaitAsync(_conn.EventTable, cancellationToken).ConfigureAwait(false); }
catch (Exception ex) { _log.LogDebug("QuestDB alarm visibility wait failed ({Exception}).", ex.GetType().Name); }
}
return outcomes;
}
private async Task<HistorianWriteOutcome> WriteOneAsync(AlarmHistorianEvent e, CancellationToken ct)
{
try
{
if (!QuestDbConnection.IsStorable(e.TimestampUtc))
{
_log.LogWarning("QuestDB alarm event dropped: its time is before 1970-01-01 and cannot be stored.");
return HistorianWriteOutcome.PermanentFail;
}
await using var conn = await _conn.OpenAsync(ct).ConfigureAwait(false);
await using var cmd = _conn.CreateCommand(conn,
$"INSERT INTO {_conn.EventTable} (ts, event_id, source, alarm_id, alarm_name, alarm_type_name, severity, event_kind, message, user_name, comment, received_utc) " +
"VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12)");
QuestDbConnection.BindTimestamp(cmd, e.TimestampUtc);
QuestDbConnection.BindText(cmd, AlarmEventIds.ComputeEventId(e));
QuestDbConnection.BindText(cmd, e.EquipmentPath ?? "");
QuestDbConnection.BindText(cmd, e.AlarmId ?? "");
QuestDbConnection.BindText(cmd, e.AlarmName ?? "");
QuestDbConnection.BindText(cmd, e.AlarmTypeName ?? "");
cmd.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Integer, Value = AlarmEventIds.ToNumericSeverity(e.Severity) });
QuestDbConnection.BindText(cmd, e.EventKind ?? "");
QuestDbConnection.BindText(cmd, e.Message);
QuestDbConnection.BindText(cmd, e.User);
QuestDbConnection.BindText(cmd, e.Comment);
QuestDbConnection.BindTimestamp(cmd, _clock.GetUtcNow().UtcDateTime);
await cmd.ExecuteNonQueryAsync(ct).ConfigureAwait(false);
return HistorianWriteOutcome.Ack;
}
catch (OperationCanceledException) when (ct.IsCancellationRequested)
{
return HistorianWriteOutcome.RetryPlease;
}
catch (Exception ex)
{
var outcome = Classify(ex);
_log.LogWarning("QuestDB alarm event write failed ({Exception}); outcome {Outcome}.", ex.GetType().Name, outcome);
return outcome;
}
}
// Only a data error QuestDB will reject forever dead-letters; anything unknown or non-database retries (never lose alarm
// history). SQLSTATE is always 00000 on QuestDB, so a server rejection is classified by message text.
internal static HistorianWriteOutcome Classify(Exception ex)
{
if (QuestDbConnection.IsTransient(ex)) return HistorianWriteOutcome.RetryPlease;
if (ex is PostgresException pg &&
PermanentMessageFragments.Any(f => pg.MessageText.Contains(f, StringComparison.OrdinalIgnoreCase)))
return HistorianWriteOutcome.PermanentFail;
return HistorianWriteOutcome.RetryPlease;
}
}
src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbConnection.cs
+247
View File
@@ -0,0 +1,247 @@
using System.Net.Sockets;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Logging.Abstractions;
using Npgsql;
using NpgsqlTypes;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb;
/// <summary>
/// Pooled Npgsql access to QuestDB plus every T0-confirmed typing rule, in one place: how timestamps are rounded and
/// bound (R6), how BYTE columns are bound and read, and how errors are classified.
/// <para>Contract for callers: bind every inbound <see cref="DateTime"/> through <see cref="BindTimestamp"/> (never
/// <c>AddWithValue</c>), call <see cref="IsStorable"/> before writing a designated timestamp, and classify
/// failures with <see cref="IsTransient"/> / <see cref="IsAlreadyExists"/>. Every QuestDB server error carries
/// SQLSTATE <c>00000</c>, so classification is by message text, never by SQLSTATE.</para>
/// </summary>
internal sealed class QuestDbConnection : IAsyncDisposable, IDisposable
{
/// <summary>
/// Npgsql pool cap applied when the connection string does not set <c>Maximum Pool Size</c> itself. Npgsql's own
/// default (100) exceeds QuestDB's default PG-wire connection limit (64), and the server's default read
/// concurrency alone can reach 64; an explicit value in the connection string is always kept.
/// </summary>
internal const int DefaultMaxPoolSize = 24;
private static readonly long MinSafeTicks = 10; // DateTime.MinValue itself binds as -infinity
private static readonly long MaxSafeTicks = DateTime.MaxValue.Ticks - DateTime.MaxValue.Ticks % 10; // MaxValue itself binds as +infinity
private readonly NpgsqlDataSource _dataSource;
/// <summary>Creates the connection helper; options must already be valid.</summary>
/// <param name="options">Validated options.</param>
/// <param name="log">Optional logger; the effective pool size is logged once at Information.</param>
public QuestDbConnection(QuestDbHistorianOptions options, ILogger? log = null)
{
ArgumentNullException.ThrowIfNull(options);
if (!QuestDbHistorianOptions.IsValidPrefix(options.TablePrefix))
throw new ArgumentException("Invalid table prefix.", nameof(options));
TablePrefix = options.TablePrefix;
CommandTimeoutSeconds = Math.Max(1, (int)Math.Ceiling(options.CommandTimeout.TotalSeconds));
var csb = new NpgsqlConnectionStringBuilder(options.ConnectionString);
var explicitPool = HasExplicitMaxPoolSize(options.ConnectionString);
if (!explicitPool) csb.MaxPoolSize = DefaultMaxPoolSize;
EffectiveMaxPoolSize = csb.MaxPoolSize;
(log ?? NullLogger.Instance).LogInformation(
"QuestDB historian connection pool: Maximum Pool Size = {MaxPoolSize} ({Source}).",
EffectiveMaxPoolSize, explicitPool ? "from the connection string" : "default cap");
_dataSource = new NpgsqlDataSourceBuilder(csb.ConnectionString).Build();
}
/// <summary>The Npgsql <c>Maximum Pool Size</c> in effect (the connection string's own value, else <see cref="DefaultMaxPoolSize"/>).</summary>
internal int EffectiveMaxPoolSize { get; }
// Npgsql accepts "Maximum Pool Size" and "MaxPoolSize" (case-insensitive); a generic parse keeps the caller's own
// keywords so "was it set" is answered by the string itself, not by the builder's default.
private static bool HasExplicitMaxPoolSize(string connectionString)
{
var generic = new System.Data.Common.DbConnectionStringBuilder { ConnectionString = connectionString };
foreach (string key in generic.Keys)
{
var k = key.Replace(" ", "", StringComparison.Ordinal);
if (string.Equals(k, "MaxPoolSize", StringComparison.OrdinalIgnoreCase)
|| string.Equals(k, "MaximumPoolSize", StringComparison.OrdinalIgnoreCase))
return true;
}
return false;
}
/// <summary>The validated table prefix.</summary>
public string TablePrefix { get; }
/// <summary>Command timeout applied by <see cref="CreateCommand"/>, in seconds.</summary>
public int CommandTimeoutSeconds { get; }
/// <summary>Prefixes a table base name (<c>sample</c>, <c>event</c>, <c>tag</c>) with the configured prefix.</summary>
/// <param name="table">Fixed base name from this project (never user input).</param>
/// <returns>The full table name.</returns>
public string T(string table) => TablePrefix + table;
/// <summary>The prefixed sample table.</summary>
public string SampleTable => T(QuestDbTables.SampleBase);
/// <summary>The prefixed event table.</summary>
public string EventTable => T(QuestDbTables.EventBase);
/// <summary>The prefixed tag table.</summary>
public string TagTable => T(QuestDbTables.TagBase);
/// <summary>Opens a pooled connection (no bootstrap; callers run the bootstrapper first).</summary>
/// <param name="ct">Cancellation.</param>
/// <returns>An open connection the caller disposes.</returns>
public async Task<NpgsqlConnection> OpenAsync(CancellationToken ct) =>
await _dataSource.OpenConnectionAsync(ct).ConfigureAwait(false);
/// <summary>Creates a text command with the configured timeout. Parameters are positional (<c>$1</c>, <c>$2</c>…).</summary>
/// <param name="conn">Open connection.</param>
/// <param name="sql">Command text.</param>
/// <returns>The command.</returns>
public NpgsqlCommand CreateCommand(NpgsqlConnection conn, string sql)
{
var cmd = conn.CreateCommand();
cmd.CommandText = sql;
cmd.CommandTimeout = CommandTimeoutSeconds;
return cmd;
}
/// <inheritdoc/>
public ValueTask DisposeAsync() => _dataSource.DisposeAsync();
/// <inheritdoc/>
public void Dispose() => _dataSource.Dispose();
// ---- timestamps (R6) -------------------------------------------------------------------------------------------
/// <summary>
/// Normalises a timestamp for QuestDB (microsecond precision). Rule: <c>Unspecified</c> is treated as UTC
/// (<c>SpecifyKind</c>); <c>Local</c> is converted with <c>ToUniversalTime</c>; then ticks are truncated to a
/// microsecond (<c>ticks - ticks % 10</c>, because Npgsql truncates relative to 2000-01-01 and so rounds UP
/// before 2000); finally the result is nudged into <c>[0001-01-01T00:00:00.000001, 9999-12-31T23:59:59.999999]</c>
/// so a value equal to <c>MinValue</c>/<c>MaxValue</c> never reaches Npgsql, which would bind it as ±infinity
/// (a <c>[Min, Max)</c> range then silently matches nothing). The result always has <c>Kind=Utc</c>.
/// </summary>
/// <param name="value">Any timestamp.</param>
/// <returns>A UTC, microsecond-aligned, finite timestamp.</returns>
public static DateTime Round(DateTime value)
{
var utc = value.Kind switch
{
DateTimeKind.Local => value.ToUniversalTime(),
DateTimeKind.Utc => value,
_ => DateTime.SpecifyKind(value, DateTimeKind.Utc),
};
var ticks = utc.Ticks - utc.Ticks % 10;
ticks = Math.Clamp(ticks, MinSafeTicks, MaxSafeTicks);
return new DateTime(ticks, DateTimeKind.Utc);
}
/// <summary>
/// True when QuestDB can store the timestamp as a designated timestamp: after <see cref="Round"/> it is on or
/// after 1970-01-01 (RC1). Writers drop (live) or refuse (HistoryUpdate) samples for which this is false; reads
/// with earlier bounds are fine.
/// </summary>
/// <param name="value">Candidate sample time.</param>
/// <returns>Whether an INSERT with this designated timestamp can succeed.</returns>
public static bool IsStorable(DateTime value) => Round(value) >= DateTime.UnixEpoch;
/// <summary>
/// Appends the next positional parameter (<c>$n</c>) as <c>timestamp with time zone</c> carrying
/// <see cref="Round"/>'s UTC value (T0-confirmed: <c>Timestamp</c>+<c>Utc</c> throws in Npgsql). It never emits
/// <c>MinValue</c>/<c>MaxValue</c>. By default a pre-1970 value is kept as is (a pre-1970 range bound is valid
/// and clamping it would move the bound across the epoch row); pass <paramref name="clampToEpoch"/> for a
/// designated-timestamp column so a pre-1970 value can only ever become the epoch itself, never a failed batch.
/// </summary>
/// <param name="cmd">Target command.</param>
/// <param name="value">Any timestamp.</param>
/// <param name="clampToEpoch">Clamp anything before 1970-01-01 to the epoch (writes only, after <see cref="IsStorable"/>).</param>
/// <returns>The added parameter.</returns>
public static NpgsqlParameter BindTimestamp(NpgsqlCommand cmd, DateTime value, bool clampToEpoch = false)
{
var rounded = Round(value);
if (clampToEpoch && rounded < DateTime.UnixEpoch) rounded = DateTime.UnixEpoch;
var p = new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.TimestampTz, Value = rounded };
cmd.Parameters.Add(p);
return p;
}
/// <summary>Appends a <c>BYTE</c> parameter (bound as <c>smallint</c>, T0-confirmed).</summary>
/// <param name="cmd">Target command.</param>
/// <param name="value">The byte.</param>
/// <returns>The added parameter.</returns>
public static NpgsqlParameter BindByte(NpgsqlCommand cmd, byte value)
{
var p = new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Smallint, Value = (short)value };
cmd.Parameters.Add(p);
return p;
}
/// <summary>Appends a <c>LONG</c> parameter (<c>bigint</c>; also the rule for integer arithmetic in tick math).</summary>
/// <param name="cmd">Target command.</param>
/// <param name="value">The value.</param>
/// <returns>The added parameter.</returns>
public static NpgsqlParameter BindLong(NpgsqlCommand cmd, long value)
{
var p = new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Bigint, Value = value };
cmd.Parameters.Add(p);
return p;
}
/// <summary>Appends a <c>SYMBOL</c>/<c>VARCHAR</c> parameter (<c>varchar</c>); null binds as NULL.</summary>
/// <param name="cmd">Target command.</param>
/// <param name="value">The string.</param>
/// <returns>The added parameter.</returns>
public static NpgsqlParameter BindText(NpgsqlCommand cmd, string? value)
{
var p = new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Varchar, Value = (object?)value ?? DBNull.Value };
cmd.Parameters.Add(p);
return p;
}
/// <summary>Reads a <c>TIMESTAMP</c> column as UTC (QuestDB returns <c>Kind=Unspecified</c> holding UTC wall clock).</summary>
/// <param name="r">Reader positioned on a row.</param>
/// <param name="i">Column ordinal.</param>
/// <returns>The timestamp with <c>Kind=Utc</c>.</returns>
public static DateTime ReadUtc(NpgsqlDataReader r, int i) =>
DateTime.SpecifyKind(r.GetDateTime(i), DateTimeKind.Utc);
/// <summary>Reads a <c>BYTE</c> column (QuestDB returns <c>smallint</c>); NULL (never stored) reads as 0 (Empty).</summary>
/// <param name="r">Reader positioned on a row.</param>
/// <param name="i">Column ordinal.</param>
/// <returns>The byte value.</returns>
public static byte ReadKind(NpgsqlDataReader r, int i) => r.IsDBNull(i) ? (byte)0 : (byte)r.GetInt16(i);
// ---- error classification --------------------------------------------------------------------------------------
/// <summary>
/// True for a transport-level or timeout failure worth retrying. A <see cref="PostgresException"/> is a
/// server-side rejection (QuestDB reports every one as SQLSTATE 00000), so it is never transient here; decide
/// those by message text.
/// </summary>
/// <param name="ex">The failure.</param>
/// <returns>Whether the failure is transient.</returns>
public static bool IsTransient(Exception ex)
{
for (var e = ex; e is not null; e = e.InnerException)
{
if (e is PostgresException or OperationCanceledException) return false;
if (e is NpgsqlException { IsTransient: true } or SocketException or IOException or TimeoutException) return true;
}
return false;
}
/// <summary>True when a server error says the object already exists (message text; SQLSTATE is always 00000).</summary>
/// <param name="ex">The failure.</param>
/// <returns>Whether it is an "already exists" rejection.</returns>
public static bool IsAlreadyExists(Exception ex) =>
ex is PostgresException pg && pg.MessageText.Contains("already exists", StringComparison.OrdinalIgnoreCase);
/// <summary>Whether a table exists, per <c>tables()</c>.</summary>
/// <param name="conn">Open connection.</param>
/// <param name="table">Full table name.</param>
/// <param name="ct">Cancellation.</param>
/// <returns>True when listed.</returns>
public async Task<bool> TableExistsAsync(NpgsqlConnection conn, string table, CancellationToken ct)
{
await using var cmd = CreateCommand(conn, "SELECT count() FROM tables() WHERE table_name = $1");
BindText(cmd, table);
return Convert.ToInt64(await cmd.ExecuteScalarAsync(ct).ConfigureAwait(false)) > 0;
}
}
src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianDataSource.cs
+484
View File
@@ -0,0 +1,484 @@
using System.Globalization;
using System.Text;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Logging.Abstractions;
using Npgsql;
using NpgsqlTypes;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb;
/// <summary>
/// QuestDB read side of the historian (spec §3 "Reads"): Raw (forward / reverse / point), AtTime (chunked
/// <c>LIMIT 1</c> neighbour seeks, T0 Q9), Events, and Processed aggregates pushed down to QuestDB SQL
/// (<see cref="QuestDbAggregateQueries"/>, T0 Q8). Results match the SQL Server data source except that every
/// timestamp is at microsecond resolution.
/// </summary>
/// <remarks>
/// <para>
/// <b>Timestamps (R6).</b> Every inbound time is normalised with <see cref="QuestDbConnection.Round"/> and every
/// <see cref="DateTime"/> is bound only through <see cref="QuestDbConnection.BindTimestamp"/>. Read bounds are not
/// clamped to the epoch (a pre-1970 bound is valid); only <c>MinValue</c>/<c>MaxValue</c> are nudged. Raw and
/// Processed results carry stored (microsecond) times. An AtTime result is evaluated at the rounded request time
/// but stamped with the client's ORIGINAL requested time (as the SQL Server backend does), so a client can match
/// each answer to its request.
/// </para>
/// <para>
/// <b>Reads never create tags (R7).</b> A tag resolves only through <see cref="QuestDbTagCatalog.ExistsAsync"/>;
/// an unknown tag is a successful empty read (Raw), an all-<c>BadNoData</c> read (AtTime / Processed), never an
/// error. Every sample query filters <c>NOT deleted</c>, so a DeleteRaw/DeleteAtTime tombstone is invisible.
/// </para>
/// <para>
/// <b>Values.</b> Every row goes through <see cref="StoredSample.FromStored"/> (an Empty row with a non-Bad status
/// is served <c>BadNoData</c>). The value kind, not which column is populated, decides the value. QuestDB stores
/// <see cref="long.MinValue"/> in a <c>LONG</c> as NULL (its NULL sentinel), so an Int64 row whose
/// <c>int_value</c> is NULL and whose <c>num_value</c> is exactly −2^63 is served as <see cref="long.MinValue"/>
/// (ruling RC2).
/// </para>
/// <para>
/// <b>Faults.</b> As in the SQL Server data source: a database / bootstrap exception is recorded in
/// <see cref="GetHealthSnapshot"/> and rethrown, failing only that call. Client-request errors — an unsupported
/// aggregate (<see cref="HistoryAggregateNotSupportedException"/>) and a processed window over
/// <see cref="BucketPlan.MaxBuckets"/> intervals (<see cref="ArgumentOutOfRangeException"/>) — are thrown before
/// any I/O and are not counted as backend failures.
/// </para>
/// </remarks>
public sealed class QuestDbHistorianDataSource : IHistorianDataSource
{
/// <inheritdoc />
/// <remarks>QuestDB stores microseconds: 10 ticks. HistoryRead paging steps off a drained timestamp by this much.</remarks>
public long TimestampResolutionTicks => 10;
/// <summary>Most AtTime request times per statement (two <c>UNION ALL</c> branches each; QuestDB 10.0.1 refuses more than 126 branches).</summary>
public const int AtTimeChunkSize = 50;
/// <summary><see cref="long.MinValue"/> as a double (−2^63, exact): what an Int64 sample of that value leaves in <c>num_value</c>.</summary>
internal const double LongNullSentinelAsDouble = -9223372036854775808d;
private const string SampleColumns = "ts, kind, num_value, int_value, str_value, status, server_ts";
private readonly QuestDbConnection _conn;
private readonly QuestDbSchemaBootstrapper _bootstrap;
private readonly QuestDbTagCatalog _catalog;
private readonly ILogger _log;
private readonly TimeProvider _time;
private readonly HistorianHealthCounters _health = new();
/// <summary>Creates the data source over the provider's shared connection, bootstrapper and tag catalog.</summary>
/// <param name="conn">Connection helper.</param>
/// <param name="bootstrap">Schema bootstrapper (shared with the other QuestDB seams).</param>
/// <param name="catalog">Tag catalog (reads use only its read-only <see cref="QuestDbTagCatalog.ExistsAsync"/>).</param>
/// <param name="log">Optional logger.</param>
/// <param name="timeProvider">Clock for "now" (AtTime beyond the last sample, server timestamps); system by default.</param>
internal QuestDbHistorianDataSource(
QuestDbConnection conn, QuestDbSchemaBootstrapper bootstrap, QuestDbTagCatalog catalog, ILogger? log = null,
TimeProvider? timeProvider = null)
{
_conn = conn ?? throw new ArgumentNullException(nameof(conn));
_bootstrap = bootstrap ?? throw new ArgumentNullException(nameof(bootstrap));
_catalog = catalog ?? throw new ArgumentNullException(nameof(catalog));
_log = log ?? NullLogger.Instance;
_time = timeProvider ?? TimeProvider.System;
}
// ------------------------------------------------------------------------------------------ Raw
/// <inheritdoc />
/// <remarks>
/// Forward (<c>start &lt; end</c>): <c>[start, end)</c> ascending. Reverse (<c>end &lt; start</c>):
/// <c>(end, start]</c> descending. <c>start == end</c>: the sample exactly at that (rounded) time.
/// <paramref name="maxValuesPerNode"/> <c>0</c> is "unbounded" (<see cref="int.MaxValue"/>); larger values clamp.
/// At most <c>top</c> samples, <c>ContinuationPoint</c> always null (R8): the node manager pages by re-reading
/// from the last returned timestamp inclusive, which the <c>(ts, tag)</c> dedup key makes exact.
/// </remarks>
public async Task<HistoryReadResult> ReadRawAsync(
string fullReference, DateTime startUtc, DateTime endUtc, uint maxValuesPerNode, CancellationToken cancellationToken)
{
ArgumentNullException.ThrowIfNull(fullReference);
var top = maxValuesPerNode == 0 ? int.MaxValue : (int)Math.Min(maxValuesPerNode, int.MaxValue);
try
{
var samples = new List<DataValueSnapshot>();
if (await ResolveAsync(fullReference, cancellationToken).ConfigureAwait(false))
{
var start = QuestDbConnection.Round(startUtc);
var end = QuestDbConnection.Round(endUtc);
string where, order;
if (start == end) { where = "ts = $2 AND ts = $3"; order = "ASC"; }
else if (start < end) { where = "ts >= $2 AND ts < $3"; order = "ASC"; }
else { where = "ts > $2 AND ts <= $3"; order = "DESC"; }
var (lo, hi) = start <= end ? (start, end) : (end, start);
await using var conn = await _conn.OpenAsync(cancellationToken).ConfigureAwait(false);
await using var cmd = _conn.CreateCommand(conn,
$"SELECT {SampleColumns} FROM {_conn.SampleTable} " +
$"WHERE tag = $1 AND {where} AND NOT deleted ORDER BY ts {order} LIMIT $4");
QuestDbConnection.BindText(cmd, fullReference);
QuestDbConnection.BindTimestamp(cmd, lo);
QuestDbConnection.BindTimestamp(cmd, hi);
cmd.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Integer, Value = top });
await using var reader = await cmd.ExecuteReaderAsync(cancellationToken).ConfigureAwait(false);
var now = _time.GetUtcNow().UtcDateTime;
while (await reader.ReadAsync(cancellationToken).ConfigureAwait(false))
{
var s = ReadStored(reader, 0);
samples.Add(new DataValueSnapshot(s.Value.ToObject(), s.Status, s.TimeUtc, ReadServerTime(reader, 6, now)));
}
}
RecordSuccess();
return new HistoryReadResult(samples, ContinuationPoint: null);
}
catch (Exception ex)
{
RecordFailure(ex);
throw;
}
}
// ------------------------------------------------------------------------------------------ AtTime
/// <inheritdoc />
/// <remarks>
/// Each distinct (rounded) requested time picks up its previous-or-equal and next live sample through two
/// <c>LIMIT 1</c> seeks, <c>UNION ALL</c>-ed, at most <see cref="AtTimeChunkSize"/> times per statement (spec §3,
/// T0 Q9). <see cref="AtTimeEvaluator.Evaluate"/> then decides each value exactly as for SQL Server (exact hit,
/// no previous ⇒ <c>BadNoData</c>, inside an outage, held before an outage, linear, stepped, after now). The
/// result matches the request in length and order (duplicates included). Each entry is evaluated at its rounded
/// time but stamped with the requested time itself (UTC-normalised as <see cref="QuestDbConnection.Round"/> does,
/// not truncated), matching the SQL Server backend. Unknown tag ⇒ every entry <c>BadNoData</c>.
/// </remarks>
public async Task<HistoryReadResult> ReadAtTimeAsync(
string fullReference, IReadOnlyList<DateTime> timestampsUtc, CancellationToken cancellationToken)
{
ArgumentNullException.ThrowIfNull(fullReference);
ArgumentNullException.ThrowIfNull(timestampsUtc);
try
{
var now = _time.GetUtcNow().UtcDateTime;
var rounded = timestampsUtc.Select(QuestDbConnection.Round).ToArray();
var byTicks = new Dictionary<long, DataValueSnapshot>(rounded.Length);
if (rounded.Length > 0 && await ResolveAsync(fullReference, cancellationToken).ConfigureAwait(false))
{
var distinct = rounded.Distinct().ToList();
await using var conn = await _conn.OpenAsync(cancellationToken).ConfigureAwait(false);
for (var offset = 0; offset < distinct.Count; offset += AtTimeChunkSize)
{
var chunk = distinct.GetRange(offset, Math.Min(AtTimeChunkSize, distinct.Count - offset));
var prev = new StoredSample?[chunk.Count];
var next = new StoredSample?[chunk.Count];
var prevServer = new DateTime?[chunk.Count];
await using (var cmd = _conn.CreateCommand(conn, AtTimeSql(_conn.SampleTable, chunk.Count)))
{
QuestDbConnection.BindText(cmd, fullReference);
foreach (var t in chunk) QuestDbConnection.BindTimestamp(cmd, t);
await using var reader = await cmd.ExecuteReaderAsync(cancellationToken).ConfigureAwait(false);
while (await reader.ReadAsync(cancellationToken).ConfigureAwait(false))
{
var k = Convert.ToInt32(reader.GetValue(0), CultureInfo.InvariantCulture);
var dir = Convert.ToInt32(reader.GetValue(1), CultureInfo.InvariantCulture);
var s = ReadStored(reader, 2);
if (dir == 0)
{
prev[k] = s;
prevServer[k] = reader.IsDBNull(8) ? null : QuestDbConnection.ReadUtc(reader, 8);
}
else
{
next[k] = s;
}
}
}
for (var k = 0; k < chunk.Count; k++)
{
var t = chunk[k];
var serverTime = prev[k] is { } ps && ps.TimeUtc == t ? prevServer[k] ?? now : now;
byTicks[t.Ticks] = AtTimeEvaluator.Evaluate(t, prev[k], next[k], serverTime, now);
}
}
}
var result = new DataValueSnapshot[rounded.Length];
for (var i = 0; i < result.Length; i++)
{
// Seek/evaluate at the rounded time; stamp the client's own requested time (M3).
var requested = AsUtc(timestampsUtc[i]);
result[i] = byTicks.TryGetValue(rounded[i].Ticks, out var v)
? v with { SourceTimestampUtc = requested }
: new DataValueSnapshot(null, HistorianStatusCodes.BadNoData, requested, now);
}
RecordSuccess();
return new HistoryReadResult(result, ContinuationPoint: null);
}
catch (Exception ex)
{
RecordFailure(ex);
throw;
}
}
/// <summary>The UTC normalisation of <see cref="QuestDbConnection.Round"/> without its truncation or nudging.</summary>
private static DateTime AsUtc(DateTime t) => t.Kind switch
{
DateTimeKind.Local => t.ToUniversalTime(),
DateTimeKind.Utc => t,
_ => DateTime.SpecifyKind(t, DateTimeKind.Utc),
};
/// <summary>
/// The T0 Q9 statement for <paramref name="count"/> request times: <c>$1</c> = tag, <c>$(k+2)</c> = time <c>k</c>.
/// Columns: <c>req, dir</c> (0 = previous-or-equal, 1 = next), then <c>ts, kind, num_value, int_value, str_value,
/// status, server_ts</c>. <c>req</c>/<c>dir</c> are generated integers, never user input.
/// </summary>
internal static string AtTimeSql(string sample, int count)
{
if (count is < 1 or > AtTimeChunkSize) throw new ArgumentOutOfRangeException(nameof(count));
var sb = new StringBuilder();
for (var k = 0; k < count; k++)
{
var p = "$" + (k + 2).ToString(CultureInfo.InvariantCulture);
var req = k.ToString(CultureInfo.InvariantCulture);
if (k > 0) sb.Append(" UNION ALL ");
sb.Append("SELECT * FROM (SELECT ").Append(req).Append(" AS req, 0 AS dir, ").Append(SampleColumns)
.Append(" FROM ").Append(sample).Append(" WHERE tag = $1 AND ts <= ").Append(p)
.Append(" AND NOT deleted ORDER BY ts DESC LIMIT 1)");
sb.Append(" UNION ALL ");
sb.Append("SELECT * FROM (SELECT ").Append(req).Append(" AS req, 1 AS dir, ").Append(SampleColumns)
.Append(" FROM ").Append(sample).Append(" WHERE tag = $1 AND ts > ").Append(p)
.Append(" AND NOT deleted ORDER BY ts ASC LIMIT 1)");
}
return sb.ToString();
}
// ------------------------------------------------------------------------------------------ Events
/// <inheritdoc />
/// <remarks>
/// <c>[start, end)</c> by event time (a reversed window is swapped), ordered by <c>(ts, event_id)</c> ascending.
/// <paramref name="maxEvents"/> &lt;= 0 or <see cref="int.MaxValue"/> ⇒ unbounded. Otherwise
/// <c>maxEvents + 1</c> rows are fetched; when the extra row arrives it is dropped and the continuation point is
/// the UTF-8 of <see cref="EventContinuation.Encode"/> for that first not-returned event.
/// </remarks>
public async Task<HistoricalEventsResult> ReadEventsAsync(
string? sourceName, DateTime startUtc, DateTime endUtc, int maxEvents, CancellationToken cancellationToken)
{
// No stored event can carry a source longer than the limit, so an over-long filter matches nothing.
if (sourceName is not null && sourceName.Length > HistorianTagNames.MaxLength)
return new HistoricalEventsResult([], null);
try
{
await _bootstrap.EnsureAsync(cancellationToken).ConfigureAwait(false);
var start = QuestDbConnection.Round(startUtc);
var end = QuestDbConnection.Round(endUtc);
var (lo, hi) = start <= end ? (start, end) : (end, start);
var bounded = maxEvents > 0 && maxEvents < int.MaxValue;
var sql =
$"SELECT event_id, source, ts, received_utc, message, severity FROM {_conn.EventTable} " +
"WHERE ts >= $1 AND ts < $2" +
(sourceName is null ? "" : " AND source = $3") +
" ORDER BY ts ASC, event_id ASC" +
(bounded ? (sourceName is null ? " LIMIT $3" : " LIMIT $4") : "");
await using var conn = await _conn.OpenAsync(cancellationToken).ConfigureAwait(false);
await using var cmd = _conn.CreateCommand(conn, sql);
QuestDbConnection.BindTimestamp(cmd, lo);
QuestDbConnection.BindTimestamp(cmd, hi);
if (sourceName is not null) QuestDbConnection.BindText(cmd, sourceName);
if (bounded) QuestDbConnection.BindLong(cmd, (long)maxEvents + 1);
var events = new List<HistoricalEvent>();
byte[]? continuation = null;
await using var reader = await cmd.ExecuteReaderAsync(cancellationToken).ConfigureAwait(false);
while (await reader.ReadAsync(cancellationToken).ConfigureAwait(false))
{
var eventTime = QuestDbConnection.ReadUtc(reader, 2);
var e = new HistoricalEvent(
reader.IsDBNull(0) ? "" : reader.GetString(0),
reader.IsDBNull(1) ? null : reader.GetString(1),
eventTime,
reader.IsDBNull(3) ? eventTime : QuestDbConnection.ReadUtc(reader, 3),
reader.IsDBNull(4) ? null : reader.GetString(4),
HistoricalEventRows.ClampSeverity(reader.IsDBNull(5) ? 0 : reader.GetInt32(5)));
if (bounded && events.Count == maxEvents)
{
continuation = Encoding.UTF8.GetBytes(EventContinuation.Encode(e.EventTimeUtc, e.EventId));
break;
}
events.Add(e);
}
RecordSuccess();
return new HistoricalEventsResult(events, continuation);
}
catch (Exception ex)
{
RecordFailure(ex);
throw;
}
}
// ------------------------------------------------------------------------------------------ Processed
/// <inheritdoc />
/// <remarks>
/// The (rounded) window is planned by <see cref="BucketPlan.Create"/>: bucket <c>i</c> =
/// <c>[start + i·interval, start + (i+1)·interval)</c>, start-inclusive / end-exclusive, the last bucket truncated
/// at <c>end</c>, interval &lt;= 0 ⇒ one bucket, <c>start == end</c> ⇒ none, a reversed window computed forward
/// and returned descending, at most <see cref="BucketPlan.MaxBuckets"/> buckets. One statement per node returns
/// the <see cref="BucketRow"/> statistics of each non-empty bucket (bucket index
/// <c>(cast(ts AS LONG)*10 − StartUnixTicks) / IntervalTicks</c>, never <c>SAMPLE BY</c>);
/// <see cref="AggregateEvaluator.Evaluate"/> produces each value and <see cref="BucketPlan.Assemble"/> fills the
/// empty buckets with <c>BadNoData</c>.
/// </remarks>
/// <exception cref="HistoryAggregateNotSupportedException">The aggregate is not one of the thirteen served.</exception>
/// <exception cref="ArgumentOutOfRangeException">The window holds more than <see cref="BucketPlan.MaxBuckets"/> intervals.</exception>
public async Task<HistoryReadResult> ReadProcessedAsync(
string fullReference, DateTime startUtc, DateTime endUtc, TimeSpan interval,
HistoryAggregateType aggregate, CancellationToken cancellationToken)
{
ArgumentNullException.ThrowIfNull(fullReference);
var family = AggregateFamilies.FamilyOf(aggregate) ?? throw new HistoryAggregateNotSupportedException(aggregate);
var plan = BucketPlan.Create(QuestDbConnection.Round(startUtc), QuestDbConnection.Round(endUtc), interval);
try
{
var now = _time.GetUtcNow().UtcDateTime;
var evaluated = new Dictionary<long, DataValueSnapshot>();
if (plan.BucketCount > 0 && await ResolveAsync(fullReference, cancellationToken).ConfigureAwait(false))
{
var sample = _conn.SampleTable;
var sql = family switch
{
AggregateFamily.Simple => QuestDbAggregateQueries.Simple(sample),
AggregateFamily.StartEnd => QuestDbAggregateQueries.StartEnd(sample),
_ => QuestDbAggregateQueries.ActualTime(sample, aggregate == HistoryAggregateType.MaximumActualTime),
};
await using var conn = await _conn.OpenAsync(cancellationToken).ConfigureAwait(false);
await using var cmd = _conn.CreateCommand(conn, sql);
QuestDbConnection.BindLong(cmd, plan.StartUnixTicks);
QuestDbConnection.BindLong(cmd, plan.IntervalTicks);
QuestDbConnection.BindText(cmd, fullReference);
QuestDbConnection.BindTimestamp(cmd, plan.StartUtc);
QuestDbConnection.BindTimestamp(cmd, plan.EndUtc);
await using var reader = await cmd.ExecuteReaderAsync(cancellationToken).ConfigureAwait(false);
while (await reader.ReadAsync(cancellationToken).ConfigureAwait(false))
{
var row = ReadBucket(reader, family);
evaluated[row.Bucket] = AggregateEvaluator.Evaluate(aggregate, row, plan.BucketStart(row.Bucket), now);
}
}
var result = plan.Assemble(evaluated, bucketStart => AggregateEvaluator.NoData(bucketStart, now));
RecordSuccess();
return new HistoryReadResult(result, ContinuationPoint: null);
}
catch (Exception ex)
{
RecordFailure(ex);
throw;
}
}
private static BucketRow ReadBucket(NpgsqlDataReader reader, AggregateFamily family)
{
static double? D(NpgsqlDataReader r, int i) => r.IsDBNull(i) ? null : r.GetDouble(i);
static int N(NpgsqlDataReader r, int i) => r.IsDBNull(i) ? 0 : Convert.ToInt32(r.GetValue(i), CultureInfo.InvariantCulture);
// A bucket without a usable sample has NULL times but kind 0 (BYTE has no NULL): absence is keyed on the time.
static HistorianValue V(NpgsqlDataReader r, int i) =>
r.IsDBNull(i + 4)
? HistorianValue.Empty
: ToValue(QuestDbConnection.ReadKind(r, i), D(r, i + 1), r.IsDBNull(i + 2) ? null : r.GetInt64(i + 2),
r.IsDBNull(i + 3) ? null : r.GetString(i + 3));
var row = new BucketRow
{
Bucket = Convert.ToInt64(reader.GetValue(0), CultureInfo.InvariantCulture),
Total = N(reader, 1),
Usable = N(reader, 2),
Good = N(reader, 3),
Bad = N(reader, 4),
NonNumeric = N(reader, 5),
};
var f = QuestDbAggregateQueries.FirstFamilyColumn;
switch (family)
{
case AggregateFamily.Simple:
row.Avg = D(reader, f);
row.Min = D(reader, f + 1);
row.Max = D(reader, f + 2);
row.Sum = D(reader, f + 3);
break;
case AggregateFamily.StartEnd:
row.First = V(reader, f);
row.Last = V(reader, f + 5);
break;
default:
row.Extreme = D(reader, f);
row.ExtremeTime = reader.IsDBNull(f + 1) ? null : QuestDbConnection.ReadUtc(reader, f + 1);
break;
}
return row;
}
// ------------------------------------------------------------------------------------------ Shared
/// <summary>
/// Reads <c>ts, kind, num_value, int_value, str_value, status</c> starting at <paramref name="o"/> and applies the
/// shared quality clamp (<see cref="StoredSample.FromStored"/>).
/// </summary>
private static StoredSample ReadStored(NpgsqlDataReader r, int o)
{
var value = ToValue(
QuestDbConnection.ReadKind(r, o + 1),
r.IsDBNull(o + 2) ? null : r.GetDouble(o + 2),
r.IsDBNull(o + 3) ? null : r.GetInt64(o + 3),
r.IsDBNull(o + 4) ? null : r.GetString(o + 4));
// status is always written; a NULL (only possible for a hand-made row) is served as "no data", never Good.
var status = r.IsDBNull(o + 5) ? HistorianStatusCodes.BadNoData : SampleColumnMapper.FromLongStatus(r.GetInt64(o + 5));
return StoredSample.FromStored(QuestDbConnection.ReadUtc(r, o), value, status);
}
/// <summary>
/// Rebuilds a value from its stored columns via the shared <see cref="SampleColumnMapper.FromColumns"/>, after the
/// RC2 repair: QuestDB reads a <c>LONG</c> holding <see cref="long.MinValue"/> back as NULL, so an Int64 row with
/// a NULL <c>int_value</c> and a <c>num_value</c> of exactly −2^63 is that value.
/// </summary>
internal static HistorianValue ToValue(byte kind, double? num, long? integer, string? str)
{
if (kind == (byte)HistorianValueKind.Int64 && integer is null && num == LongNullSentinelAsDouble)
integer = long.MinValue;
return SampleColumnMapper.FromColumns(kind, num, integer, str);
}
private static DateTime ReadServerTime(NpgsqlDataReader r, int i, DateTime fallback) =>
r.IsDBNull(i) ? fallback : QuestDbConnection.ReadUtc(r, i);
/// <summary>Read-only tag lookup (R7; never creates a row); runs the schema bootstrap on a cache miss.</summary>
private Task<bool> ResolveAsync(string name, CancellationToken ct) => _catalog.ExistsAsync(name, ct);
// ------------------------------------------------------------------------------------------ Health
/// <inheritdoc />
public HistorianHealthSnapshot GetHealthSnapshot() => _health.Snapshot();
private void RecordSuccess() => _health.RecordSuccess();
private void RecordFailure(Exception ex)
{
// A cancelled read (deadline / client abort) is not a backend fault (the counters ignore it too).
if (ex is OperationCanceledException) return;
_health.RecordFailure(ex);
_log.LogDebug("QuestDB historian read failed ({ExceptionType})", ex.GetType().Name);
}
/// <inheritdoc />
public void Dispose()
{
// Connections are per call (pooled by the shared NpgsqlDataSource, owned by the provider); nothing to release.
}
}
src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianDataUpdater.cs
+289
View File
@@ -0,0 +1,289 @@
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Logging.Abstractions;
using Npgsql;
using NpgsqlTypes;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb;
/// <summary>
/// OPC UA HistoryUpdate against the QuestDB sample table (spec §2.3). Every operation is an <c>INSERT</c>: the
/// table's <c>DEDUP UPSERT KEYS(ts, tag)</c> performs the replace, and a delete writes a tombstone
/// (<c>deleted = true</c>, NULL values, <c>BadNoData</c>). QuestDB <c>UPDATE</c>/<c>DELETE</c> are never issued.
/// <para>Every status comes from <see cref="HistoryUpdatePlanner"/>, so codes equal the SQL Server backend's.
/// "Exists" means present and not deleted, so Insert over a tombstone succeeds (<c>GoodEntryInserted</c>) while
/// Replace and DeleteAtTime treat a tombstone as no entry. Existence is read with one range query and filtered
/// client-side (never an IN list).</para>
/// <para><b>Concurrency:</b> calls are serialized per tag inside this instance (64 striped async locks held from
/// the existence read through the visibility wait), so concurrent Insert/Replace/Delete on one tag cannot
/// both act on a stale read. This does NOT cover the live writer racing on the same key, nor a second node;
/// that residual is accepted.</para>
/// <para><b>Pre-1970 (RC1):</b> QuestDB cannot store such a designated timestamp. Insert/Replace/Update of such an
/// item reports <c>BadOutOfRange</c> and sends nothing (a null/<c>MinValue</c> timestamp is still
/// <c>BadInvalidTimestamp</c>); DeleteAtTime reports <c>BadNoEntryExists</c>; a DeleteRaw window may start before
/// 1970 (read bounds are not clamped).</para>
/// <para><b>Read-your-writes:</b> after writing, <see cref="QuestDbWalVisibility"/> waits for the rows to become
/// visible, bounded by <c>VisibilityTimeout</c>; a timeout only logs a Warning, never changes a code.</para>
/// </summary>
internal sealed class QuestDbHistorianDataUpdater : IHistorianDataUpdater
{
/// <summary>Rows per INSERT and per DeleteRaw page (T0 Q2: 9,000 parameters, far under the 32,767 limit).</summary>
internal const int BatchSize = 1000;
private const long ResolutionTicks = 10;
private const int LockStripes = 64;
private readonly SemaphoreSlim[] _locks = Enumerable.Range(0, LockStripes).Select(_ => new SemaphoreSlim(1, 1)).ToArray();
private long _liveRowsRead;
private readonly QuestDbConnection _conn;
private readonly QuestDbTagCatalog _tags;
private readonly QuestDbWalVisibility _wal;
private readonly ILogger _log;
/// <summary>Creates the updater.</summary>
/// <param name="conn">Connection helper.</param>
/// <param name="tags">Tag catalog (R7: the only known-tag authority).</param>
/// <param name="wal">WAL visibility wait (R5).</param>
/// <param name="log">Optional logger.</param>
public QuestDbHistorianDataUpdater(QuestDbConnection conn, QuestDbTagCatalog tags, QuestDbWalVisibility wal,
ILogger? log = null)
{
_conn = conn;
_tags = tags;
_wal = wal;
_log = log ?? NullLogger.Instance;
}
/// <summary>Rows returned by existence reads so far (test/diagnostic counter).</summary>
internal long LiveRowsRead => Interlocked.Read(ref _liveRowsRead);
private async Task<IDisposable> LockAsync(string tag, CancellationToken ct)
{
var sem = _locks[(int)((uint)StringComparer.Ordinal.GetHashCode(tag) % LockStripes)];
await sem.WaitAsync(ct).ConfigureAwait(false);
return new Releaser(sem);
}
private sealed class Releaser(SemaphoreSlim sem) : IDisposable
{
public void Dispose() => sem.Release();
}
/// <inheritdoc/>
public async Task<IReadOnlyList<uint>> UpdateDataAsync(string tag, HistoryUpdateMode mode,
IReadOnlyList<HistorizationValue> values, CancellationToken ct)
{
ArgumentNullException.ThrowIfNull(tag);
ArgumentNullException.ThrowIfNull(values);
// RC1: a valid-timestamp item before 1970 can never be stored; refuse it and plan the rest.
var final = new uint[values.Count];
var kept = new List<int>(values.Count);
for (var i = 0; i < values.Count; i++)
{
if (values[i].TimestampUtc is { } ts && ts != DateTime.MinValue && !QuestDbConnection.IsStorable(ts))
final[i] = HistorianStatusCodes.BadOutOfRange;
else
kept.Add(i);
}
var plan = HistoryUpdatePlanner.Plan(tag, mode, kept.Select(i => values[i]).ToList(), ResolutionTicks);
IReadOnlyList<uint> planned;
if (plan.Winners.Count == 0)
{
planned = HistoryUpdatePlanner.Complete(plan, mode, new Dictionary<long, UpsertOutcome>());
}
else
{
using var _ = await LockAsync(tag, ct).ConfigureAwait(false);
var known = mode == HistoryUpdateMode.Replace
? await _tags.ExistsAsync(tag, ct).ConfigureAwait(false)
: await EnsureAsync(tag, plan.Winners[0].Row.Value.Kind, ct).ConfigureAwait(false);
var outcome = known
? await ApplyAsync(tag, mode, plan.Winners.Select(w => w.Row).ToList(), ct).ConfigureAwait(false)
: new Dictionary<long, UpsertOutcome>();
planned = HistoryUpdatePlanner.Complete(plan, mode, outcome);
}
for (var k = 0; k < kept.Count; k++) final[kept[k]] = planned[k];
return final;
}
/// <inheritdoc/>
public async Task<uint> DeleteRawAsync(string tag, DateTime startUtc, DateTime endUtc, CancellationToken ct)
{
ArgumentNullException.ThrowIfNull(tag);
if (!HistorianTagNames.IsValid(tag)) return HistorianStatusCodes.BadInvalidArgument;
if (!await _tags.ExistsAsync(tag, ct).ConfigureAwait(false))
return HistoryUpdatePlanner.DeleteRawResult(tagKnown: false, 0);
using var _ = await LockAsync(tag, ct).ConfigureAwait(false);
var cursor = QuestDbConnection.Round(startUtc);
var end = QuestDbConnection.Round(endUtc);
long total = 0;
while (cursor < end)
{
var page = new List<DateTime>(BatchSize);
await using (var conn = await _conn.OpenAsync(ct).ConfigureAwait(false))
await using (var cmd = _conn.CreateCommand(conn,
$"SELECT ts FROM {_conn.SampleTable} WHERE tag = $1 AND ts >= $2 AND ts < $3 AND NOT deleted ORDER BY ts LIMIT {BatchSize}"))
{
QuestDbConnection.BindText(cmd, tag);
QuestDbConnection.BindTimestamp(cmd, cursor);
QuestDbConnection.BindTimestamp(cmd, end);
await using var rd = await cmd.ExecuteReaderAsync(ct).ConfigureAwait(false);
while (await rd.ReadAsync(ct).ConfigureAwait(false)) page.Add(QuestDbConnection.ReadUtc(rd, 0));
}
if (page.Count == 0) break;
await WriteTombstonesAsync(tag, page, ct).ConfigureAwait(false);
// The wait between pages is required: without it the next page could re-read rows just tombstoned.
await _wal.WaitAsync(_conn.SampleTable, ct).ConfigureAwait(false);
total += page.Count;
if (page.Count < BatchSize) break;
cursor = page[^1].AddTicks(ResolutionTicks);
}
return HistoryUpdatePlanner.DeleteRawResult(tagKnown: true, total);
}
/// <inheritdoc/>
public async Task<IReadOnlyList<uint>> DeleteAtTimeAsync(string tag, IReadOnlyList<DateTime> timesUtc, CancellationToken ct)
{
ArgumentNullException.ThrowIfNull(tag);
ArgumentNullException.ThrowIfNull(timesUtc);
if (!HistorianTagNames.IsValid(tag))
{
var refused = new uint[timesUtc.Count];
Array.Fill(refused, HistorianStatusCodes.BadInvalidArgument);
return refused;
}
// M-2: round BEFORE both the delete and the comparison, so a sub-µs request time matches the row it deleted.
// MinValue stays as is: the planner turns it into BadInvalidTimestamp.
var rounded = timesUtc.Select(t => t == DateTime.MinValue ? t : QuestDbConnection.Round(t)).ToList();
var candidates = new HashSet<long>();
foreach (var t in rounded)
if (t != DateTime.MinValue && t >= DateTime.UnixEpoch) candidates.Add(t.Ticks); // pre-1970 can hold no row
var deleted = new HashSet<long>();
if (candidates.Count == 0 || !await _tags.ExistsAsync(tag, ct).ConfigureAwait(false))
return HistoryUpdatePlanner.DeleteAtTimeResults(rounded, deleted, tagKnown: false);
using var _ = await LockAsync(tag, ct).ConfigureAwait(false);
var live = await ReadLiveAsync(tag, candidates.Select(t => new DateTime(t, DateTimeKind.Utc)).ToList(), ct).ConfigureAwait(false);
var toDelete = candidates.Where(live.Contains).OrderBy(x => x).ToList();
foreach (var chunk in toDelete.Chunk(BatchSize))
await WriteTombstonesAsync(tag, chunk.Select(t => new DateTime(t, DateTimeKind.Utc)).ToList(), ct).ConfigureAwait(false);
if (toDelete.Count > 0) await _wal.WaitAsync(_conn.SampleTable, ct).ConfigureAwait(false);
deleted.UnionWith(toDelete);
return HistoryUpdatePlanner.DeleteAtTimeResults(rounded, deleted, tagKnown: true);
}
private async Task<bool> EnsureAsync(string tag, HistorianValueKind kind, CancellationToken ct)
{
await _tags.EnsureAsync(tag, HistorianDataTypes.FromKind(kind), ct).ConfigureAwait(false);
return true;
}
private async Task<Dictionary<long, UpsertOutcome>> ApplyAsync(string tag, HistoryUpdateMode mode,
List<HistorianSampleRow> rows, CancellationToken ct)
{
// Key = the planner's tick (what Complete looks up); the stored time is Round(of the same row).
var live = await ReadLiveAsync(tag, rows.Select(r => QuestDbConnection.Round(r.TimeUtc)).ToList(), ct).ConfigureAwait(false);
var outcome = new Dictionary<long, UpsertOutcome>();
var toWrite = new List<HistorianSampleRow>();
foreach (var r in rows)
{
var present = live.Contains(QuestDbConnection.Round(r.TimeUtc).Ticks);
if (mode == HistoryUpdateMode.Insert && present) continue;
if (mode == HistoryUpdateMode.Replace && !present) continue;
outcome[r.TimeUtc.Ticks] = present ? UpsertOutcome.Replaced : UpsertOutcome.Inserted;
toWrite.Add(r);
}
foreach (var chunk in toWrite.Chunk(BatchSize))
await WriteRowsAsync(tag, chunk, ct).ConfigureAwait(false);
if (toWrite.Count > 0) await _wal.WaitAsync(_conn.SampleTable, ct).ConfigureAwait(false);
return outcome;
}
/// <summary>Ticks of the live (present, not deleted) rows near the given times: one range query per cluster (I-2).</summary>
private async Task<HashSet<long>> ReadLiveAsync(string tag, IReadOnlyList<DateTime> times, CancellationToken ct)
{
var live = new HashSet<long>();
await using var conn = await _conn.OpenAsync(ct).ConfigureAwait(false);
foreach (var (lo, hi) in QuestDbTimeClusters.Cluster(times, QuestDbTimeClusters.ExistenceGap))
{
await using var cmd = _conn.CreateCommand(conn,
$"SELECT ts FROM {_conn.SampleTable} WHERE tag = $1 AND ts >= $2 AND ts <= $3 AND NOT deleted");
QuestDbConnection.BindText(cmd, tag);
QuestDbConnection.BindTimestamp(cmd, lo);
QuestDbConnection.BindTimestamp(cmd, hi);
await using var rd = await cmd.ExecuteReaderAsync(ct).ConfigureAwait(false);
while (await rd.ReadAsync(ct).ConfigureAwait(false))
{
live.Add(QuestDbConnection.ReadUtc(rd, 0).Ticks);
Interlocked.Increment(ref _liveRowsRead);
}
}
return live;
}
private Task WriteRowsAsync(string tag, IReadOnlyList<HistorianSampleRow> rows, CancellationToken ct) =>
InsertAsync(rows.Count, ct, (cmd, i) =>
{
var row = rows[i];
var c = SampleColumnMapper.ToColumns(row.Value);
QuestDbConnection.BindTimestamp(cmd, row.TimeUtc, clampToEpoch: true);
QuestDbConnection.BindText(cmd, tag);
QuestDbConnection.BindByte(cmd, c.ValueKind);
Typed(cmd, NpgsqlDbType.Double, c.NumValue);
Typed(cmd, NpgsqlDbType.Bigint, c.IntValue);
Typed(cmd, NpgsqlDbType.Varchar, c.StrValue);
QuestDbConnection.BindLong(cmd, SampleColumnMapper.ToLongStatus(row.StatusCode));
Typed(cmd, NpgsqlDbType.Boolean, false);
});
private Task WriteTombstonesAsync(string tag, IReadOnlyList<DateTime> times, CancellationToken ct) =>
InsertAsync(times.Count, ct, (cmd, i) =>
{
QuestDbConnection.BindTimestamp(cmd, times[i], clampToEpoch: true);
QuestDbConnection.BindText(cmd, tag);
QuestDbConnection.BindByte(cmd, (byte)HistorianValueKind.Empty);
Typed(cmd, NpgsqlDbType.Double, null);
Typed(cmd, NpgsqlDbType.Bigint, null);
Typed(cmd, NpgsqlDbType.Varchar, null);
QuestDbConnection.BindLong(cmd, SampleColumnMapper.ToLongStatus(HistorianStatusCodes.BadNoData));
Typed(cmd, NpgsqlDbType.Boolean, true);
});
/// <summary>Multi-row INSERT; <paramref name="bindRow"/> adds the first eight parameters of a row, this adds server_ts.</summary>
private async Task InsertAsync(int count, CancellationToken ct, Action<NpgsqlCommand, int> bindRow)
{
if (count == 0) return;
await using var conn = await _conn.OpenAsync(ct).ConfigureAwait(false);
await using var cmd = _conn.CreateCommand(conn, "");
var now = DateTime.UtcNow;
var sb = new System.Text.StringBuilder(
$"INSERT INTO {_conn.SampleTable} (ts, tag, kind, num_value, int_value, str_value, status, deleted, server_ts) VALUES ");
for (var i = 0; i < count; i++)
{
bindRow(cmd, i);
QuestDbConnection.BindTimestamp(cmd, now);
var b = i * 9;
if (i > 0) sb.Append(',');
sb.Append('(');
for (var p = 1; p <= 9; p++) sb.Append(p > 1 ? "," : "").Append('$').Append(b + p);
sb.Append(')');
}
cmd.CommandText = sb.ToString();
try
{
await cmd.ExecuteNonQueryAsync(ct).ConfigureAwait(false);
}
catch (Exception ex) when (ex is not OperationCanceledException)
{
_log.LogWarning(ex, "QuestDB HistoryUpdate INSERT of {Count} rows into {Table} failed.", count, _conn.SampleTable);
throw;
}
}
private static void Typed(NpgsqlCommand cmd, NpgsqlDbType type, object? value) =>
cmd.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = type, Value = value ?? DBNull.Value });
}
src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianOptions.cs
+57
View File
@@ -0,0 +1,57 @@
using System.Text.RegularExpressions;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb;
/// <summary>Options for the QuestDB historian backend (the <c>ServerHistorian:QuestDb</c> config block).</summary>
public sealed partial class QuestDbHistorianOptions
{
/// <summary>
/// Required. An Npgsql connection string for QuestDB's PG-wire port, e.g.
/// <c>Host=questdb;Port=8812;Username=admin;Password=quest;Database=qdb;ServerCompatibilityMode=NoTypeLoading</c>.
/// </summary>
public string ConnectionString { get; set; } = "";
/// <summary>Prefix of the three tables (QuestDB has no schemas). Spliced into DDL/DML, so it is regex-validated.</summary>
public string TablePrefix { get; set; } = "hist_";
/// <summary>Retention in days applied as a TTL to the sample and event tables; <c>null</c> keeps data forever.</summary>
public int? RetentionDays { get; set; }
/// <summary>Per-command timeout. Must be positive.</summary>
public TimeSpan CommandTimeout { get; set; } = TimeSpan.FromSeconds(30);
/// <summary>
/// <c>true</c> creates the tables (and applies the TTL) on first use; <c>false</c> only verifies they exist and
/// fails every historian call with <c>HistorianSchemaMissingException</c> otherwise.
/// </summary>
public bool AutoCreateSchema { get; set; } = true;
/// <summary>How long a write waits for its rows to become visible to queries (WAL apply). 0 disables the wait.</summary>
public TimeSpan VisibilityTimeout { get; set; } = TimeSpan.FromSeconds(5);
[GeneratedRegex("^[A-Za-z0-9_]+$")]
private static partial Regex PrefixRegex();
/// <summary>True when <paramref name="prefix"/> is safe to splice into a table name.</summary>
/// <param name="prefix">Candidate prefix.</param>
/// <returns>Whether it matches <c>^[A-Za-z0-9_]+$</c>.</returns>
public static bool IsValidPrefix(string? prefix) => prefix is not null && PrefixRegex().IsMatch(prefix);
/// <summary>Validates the options without touching QuestDB.</summary>
/// <returns>One message per problem (each names its <c>ServerHistorian:QuestDb:&lt;Key&gt;</c>); empty when valid.</returns>
public IReadOnlyList<string> Validate()
{
var errors = new List<string>();
if (string.IsNullOrWhiteSpace(ConnectionString))
errors.Add("ServerHistorian:QuestDb:ConnectionString is required.");
if (!IsValidPrefix(TablePrefix))
errors.Add("ServerHistorian:QuestDb:TablePrefix must match ^[A-Za-z0-9_]+$.");
if (RetentionDays is < 1)
errors.Add("ServerHistorian:QuestDb:RetentionDays must be at least 1 when set.");
if (CommandTimeout <= TimeSpan.Zero)
errors.Add("ServerHistorian:QuestDb:CommandTimeout must be greater than zero.");
if (VisibilityTimeout < TimeSpan.Zero)
errors.Add("ServerHistorian:QuestDb:VisibilityTimeout must not be negative (0 disables the wait).");
return errors;
}
}
src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianProvider.cs
+132
View File
@@ -0,0 +1,132 @@
using System.Collections.Frozen;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Logging.Abstractions;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian;
using ZB.MOM.WW.OtOpcUa.Runtime.Historian;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb;
/// <summary>
/// <see cref="IHistorianBackendProvider"/> for the QuestDB historian. Supports every
/// <see cref="HistorianValueKind"/> and HistoryUpdate (soft delete). All seams share ONE connection helper, ONE
/// schema bootstrapper, ONE tag catalog (R7: the only "known tag" authority) and ONE WAL visibility wait, so the
/// schema is ensured once and known tags are cached once.
/// </summary>
/// <remarks>
/// The shared state is built on the first <c>Create*</c> call, not in the constructor: the constructor does no
/// validation and no I/O, so resolving the provider for a backend nobody consumes never fails on its options
/// (they are validated, consumer-gated, by the host's <c>ServerHistorianOptionsValidator</c>). The first
/// <c>Create*</c> call's service provider supplies the logger factory the shared pieces log through.
/// <para>The provider owns the shared connection pool and disposes it (IAsyncDisposable/IDisposable); the host
/// registers the provider through a DI factory so the container disposes it at shutdown.</para>
/// </remarks>
public sealed class QuestDbHistorianProvider : IHistorianBackendProvider, IAsyncDisposable, IDisposable
{
private static readonly FrozenSet<HistorianValueKind> Kinds =
Enum.GetValues<HistorianValueKind>().ToFrozenSet();
private readonly QuestDbHistorianOptions _options;
private readonly object _initLock = new();
private Shared? _shared;
private bool _disposed;
/// <summary>Creates the provider over the <c>ServerHistorian:QuestDb</c> options (validated elsewhere).</summary>
/// <param name="options">The <c>ServerHistorian:QuestDb</c> options.</param>
public QuestDbHistorianProvider(QuestDbHistorianOptions options)
{
_options = options ?? throw new ArgumentNullException(nameof(options));
}
/// <inheritdoc />
public string Name => "QuestDb";
/// <inheritdoc />
public IReadOnlySet<HistorianValueKind> SupportedKinds => Kinds;
private static ILoggerFactory Loggers(IServiceProvider sp) =>
(sp.GetService(typeof(ILoggerFactory)) as ILoggerFactory) ?? NullLoggerFactory.Instance;
private Shared Get(IServiceProvider sp)
{
if (Volatile.Read(ref _shared) is { } built) return built;
lock (_initLock)
{
if (_shared is { } raced) return raced;
var loggers = Loggers(sp);
ObjectDisposedException.ThrowIf(_disposed, this);
var conn = new QuestDbConnection(_options, loggers.CreateLogger<QuestDbConnection>());
var bootstrap = new QuestDbSchemaBootstrapper(conn, _options, loggers.CreateLogger<QuestDbSchemaBootstrapper>());
var wal = new QuestDbWalVisibility(conn, _options.VisibilityTimeout, loggers.CreateLogger<QuestDbWalVisibility>());
var shared = new Shared(conn, bootstrap, new QuestDbTagCatalog(conn, bootstrap, wal), wal);
Volatile.Write(ref _shared, shared);
return shared;
}
}
/// <inheritdoc />
public IHistorianDataSource CreateDataSource(IServiceProvider sp)
{
var s = Get(sp);
return new QuestDbHistorianDataSource(s.Conn, s.Bootstrap, s.Catalog,
Loggers(sp).CreateLogger<QuestDbHistorianDataSource>());
}
/// <inheritdoc />
public IHistorianValueWriter CreateValueWriter(IServiceProvider sp)
{
var s = Get(sp);
return new QuestDbHistorianValueWriter(s.Conn, s.Catalog, s.Wal,
Loggers(sp).CreateLogger<QuestDbHistorianValueWriter>());
}
/// <inheritdoc />
public IAlarmHistorianWriter CreateAlarmWriter(IServiceProvider sp)
{
var s = Get(sp);
return new QuestDbAlarmHistorianWriter(s.Conn, s.Bootstrap, s.Wal,
Loggers(sp).CreateLogger<QuestDbAlarmHistorianWriter>());
}
/// <inheritdoc />
public IHistorianProvisioning CreateProvisioning(IServiceProvider sp)
{
var s = Get(sp);
return new QuestDbTagProvisioner(s.Conn, s.Bootstrap, s.Catalog, s.Wal,
Loggers(sp).CreateLogger<QuestDbTagProvisioner>());
}
/// <inheritdoc />
/// <remarks>The soft-delete updater: deletes write tombstones (INSERT only, never QuestDB UPDATE/DELETE).</remarks>
public IHistorianDataUpdater? CreateUpdater(IServiceProvider sp)
{
var s = Get(sp);
return new QuestDbHistorianDataUpdater(s.Conn, s.Catalog, s.Wal,
Loggers(sp).CreateLogger<QuestDbHistorianDataUpdater>());
}
/// <summary>Disposes the shared connection pool, when one was built. Idempotent.</summary>
/// <returns>A task completing when the pool is disposed.</returns>
public async ValueTask DisposeAsync()
{
if (TakeForDispose() is { } conn) await conn.DisposeAsync().ConfigureAwait(false);
}
/// <summary>Disposes the shared connection pool, when one was built. Idempotent.</summary>
public void Dispose() => TakeForDispose()?.Dispose();
private QuestDbConnection? TakeForDispose()
{
lock (_initLock)
{
if (_disposed) return null;
_disposed = true;
var conn = _shared?.Conn;
_shared = null;
return conn;
}
}
private sealed record Shared(
QuestDbConnection Conn, QuestDbSchemaBootstrapper Bootstrap, QuestDbTagCatalog Catalog, QuestDbWalVisibility Wal);
}
src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianValueWriter.cs
+170
View File
@@ -0,0 +1,170 @@
using System.Collections.Concurrent;
using System.Text;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Logging.Abstractions;
using Npgsql;
using NpgsqlTypes;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb;
/// <summary>
/// <see cref="IHistorianValueWriter"/> over the QuestDB sample table. Redelivery-idempotent with first-wins
/// semantics: QuestDB's <c>DEDUP UPSERT KEYS(ts, tag)</c> would let a re-sent sample overwrite the stored one (and a
/// re-sent live sample resurrect a tombstone), so the writer reads which timestamps already exist for the tag in
/// the batch's range (tombstones count as present, T0 Q11 range form) and inserts only the absent rows.
/// Never throws; returns false on any failure so the recorder keeps its entries.
/// </summary>
/// <remarks>
/// Batch normalisation is <see cref="LiveWritePreparer.Prepare"/> at µs resolution (non-finite doubles become Empty
/// + Bad, an in-batch µs collision keeps the first). <b>RC1:</b> a sample whose time is before 1970 cannot be a
/// QuestDB designated timestamp and one such row would fail the whole multi-row INSERT, so those samples are dropped
/// (Warning at most once per tag per <see cref="Pre1970WarningInterval"/>; every one counted in <see cref="DroppedPre1970"/>) and the call still succeeds for the rest.
/// <b>R5:</b> after inserting, the writer waits for WAL visibility so an immediate redelivery sees the rows.
/// Inserts are chunked at <see cref="BatchRows"/> rows (8 bound parameters per row, far under the 32,767 limit).
/// </remarks>
internal sealed class QuestDbHistorianValueWriter : IHistorianValueWriter
{
/// <summary>Rows per multi-row INSERT (T0 Q2: 1,000 rows is reliable).</summary>
internal const int BatchRows = 1000;
/// <summary>
/// Minimum time between two pre-1970 drop Warnings for the same tag. A device with an unset clock sends such
/// times on every drain pass; <see cref="DroppedPre1970"/> still counts every dropped sample.
/// </summary>
internal static readonly TimeSpan Pre1970WarningInterval = TimeSpan.FromMinutes(10);
private readonly QuestDbConnection _conn;
private readonly QuestDbTagCatalog _catalog;
private readonly QuestDbWalVisibility _wal;
private readonly ILogger _log;
private readonly Func<DateTime> _utcNow;
private long _droppedPre1970;
private readonly ConcurrentDictionary<string, DateTime> _pre1970WarnedUtc = new(StringComparer.Ordinal);
/// <summary>Creates the writer.</summary>
/// <param name="conn">Connection helper.</param>
/// <param name="catalog">Tag catalog (the only tag authority).</param>
/// <param name="wal">WAL visibility wait.</param>
/// <param name="log">Optional logger.</param>
/// <param name="utcNow">Clock for null timestamps and <c>server_ts</c> (tests).</param>
public QuestDbHistorianValueWriter(QuestDbConnection conn, QuestDbTagCatalog catalog, QuestDbWalVisibility wal,
ILogger? log = null, Func<DateTime>? utcNow = null)
{
_conn = conn;
_catalog = catalog;
_wal = wal;
_log = log ?? NullLogger.Instance;
_utcNow = utcNow ?? (() => DateTime.UtcNow);
}
/// <summary>Samples dropped so far because their time is before 1970 (RC1).</summary>
internal long DroppedPre1970 => Interlocked.Read(ref _droppedPre1970);
/// <inheritdoc/>
public async Task<bool> WriteLiveValuesAsync(string tag, IReadOnlyList<HistorizationValue> values, CancellationToken ct)
{
try
{
if (!HistorianTagNames.IsValid(tag))
{
_log.LogWarning("QuestDB historian live-value write refused: tag name is missing or longer than {Max} characters.",
HistorianTagNames.MaxLength);
return false;
}
if (values is null || values.Count == 0) return true;
var prepared = LiveWritePreparer.Prepare(values, _utcNow, resolutionTicks: 10);
var rows = prepared.Where(r => QuestDbConnection.IsStorable(r.TimeUtc)).ToList();
var dropped = prepared.Count - rows.Count;
if (dropped > 0)
{
Interlocked.Add(ref _droppedPre1970, dropped);
if (ShouldWarnPre1970(tag))
_log.LogWarning("QuestDB historian dropped {Count} sample(s) for a tag because their time is before 1970-01-01 (not storable); "
+ "further drops for this tag are not logged for {Minutes} minutes.", dropped, Pre1970WarningInterval.TotalMinutes);
}
if (rows.Count == 0) return true;
var dataType = HistorianDataTypes.FromKind(rows.Select(r => r.Value.Kind).FirstOrDefault(k => k != HistorianValueKind.Empty));
await _catalog.EnsureAsync(tag, dataType, ct).ConfigureAwait(false);
var present = await ExistingTicksAsync(tag, rows, ct).ConfigureAwait(false);
var fresh = rows.Where(r => !present.Contains(QuestDbConnection.Round(r.TimeUtc).Ticks)).ToList();
if (fresh.Count == 0) return true;
var serverTs = _utcNow();
for (var i = 0; i < fresh.Count; i += BatchRows)
await InsertAsync(tag, fresh.GetRange(i, Math.Min(BatchRows, fresh.Count - i)), serverTs, ct).ConfigureAwait(false);
await _wal.WaitAsync(_conn.SampleTable, ct).ConfigureAwait(false); // timeout/suspension does not change the result (R5)
return true;
}
catch (Exception ex)
{
_log.LogWarning("QuestDB historian live-value write failed ({Exception}); will retry.", ex.GetType().Name);
return false;
}
}
// Once per tag per Pre1970WarningInterval, measured on the writer's clock.
private bool ShouldWarnPre1970(string tag)
{
var now = _utcNow();
var warn = false;
_pre1970WarnedUtc.AddOrUpdate(tag,
_ => { warn = true; return now; },
(_, last) =>
{
if (now - last < Pre1970WarningInterval) { warn = false; return last; }
warn = true;
return now;
});
return warn;
}
// T0 Q11 range form, one query per cluster: tombstones are NOT filtered out (a deleted row must not be resurrected).
private async Task<HashSet<long>> ExistingTicksAsync(string tag, List<HistorianSampleRow> rows, CancellationToken ct)
{
var times = rows.Select(r => QuestDbConnection.Round(r.TimeUtc)).OrderBy(t => t).ToList();
var set = new HashSet<long>();
await using var conn = await _conn.OpenAsync(ct).ConfigureAwait(false);
foreach (var (lo, hi) in QuestDbTimeClusters.Cluster(times, QuestDbTimeClusters.ExistenceGap, BatchRows))
{
await using var cmd = _conn.CreateCommand(conn, $"SELECT ts FROM {_conn.SampleTable} WHERE tag = $1 AND ts >= $2 AND ts <= $3");
QuestDbConnection.BindText(cmd, tag);
QuestDbConnection.BindTimestamp(cmd, lo);
QuestDbConnection.BindTimestamp(cmd, hi);
await using var r = await cmd.ExecuteReaderAsync(ct).ConfigureAwait(false);
while (await r.ReadAsync(ct).ConfigureAwait(false)) set.Add(QuestDbConnection.ReadUtc(r, 0).Ticks);
}
return set;
}
private async Task InsertAsync(string tag, List<HistorianSampleRow> chunk, DateTime serverTs, CancellationToken ct)
{
var sql = new StringBuilder($"INSERT INTO {_conn.SampleTable} (ts, tag, kind, num_value, int_value, str_value, status, deleted, server_ts) VALUES ");
await using var conn = await _conn.OpenAsync(ct).ConfigureAwait(false);
await using var cmd = _conn.CreateCommand(conn, "");
var n = 1;
for (var i = 0; i < chunk.Count; i++)
{
if (i > 0) sql.Append(',');
sql.Append($"(${n},${n + 1},${n + 2},${n + 3},${n + 4},${n + 5},${n + 6},false,${n + 7})");
n += 8;
var row = chunk[i];
var c = SampleColumnMapper.ToColumns(row.Value);
QuestDbConnection.BindTimestamp(cmd, row.TimeUtc);
QuestDbConnection.BindText(cmd, tag);
QuestDbConnection.BindByte(cmd, c.ValueKind);
cmd.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Double, Value = c.NumValue is { } d ? d : DBNull.Value });
cmd.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Bigint, Value = c.IntValue is { } l ? l : DBNull.Value });
QuestDbConnection.BindText(cmd, c.StrValue);
QuestDbConnection.BindLong(cmd, SampleColumnMapper.ToLongStatus(row.StatusCode));
QuestDbConnection.BindTimestamp(cmd, serverTs);
}
cmd.CommandText = sql.ToString();
await cmd.ExecuteNonQueryAsync(ct).ConfigureAwait(false);
}
}
src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbSchemaBootstrapper.cs
+143
View File
@@ -0,0 +1,143 @@
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Logging.Abstractions;
using Npgsql;
using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb;
/// <summary>
/// Lazily creates the three tables once per instance (<c>CREATE TABLE IF NOT EXISTS</c>), then reconciles the TTL
/// of the sample and event tables with <see cref="QuestDbHistorianOptions.RetentionDays"/>.
/// <list type="bullet">
/// <item><b>Concurrency:</b> two nodes starting together are safe. If a CREATE still fails with a server error
/// whose message says <c>already exists</c>, <c>tables()</c> is re-checked and the error is swallowed only when the
/// table is really there; anything else is rethrown (SQLSTATE is always <c>00000</c>, so it cannot classify).</item>
/// <item><b>TTL (RC3):</b> <c>tables()</c> normalises units (7 days reads back as <c>1 WEEK</c>) so values are compared
/// in hours, and an ALTER becomes visible only after the WAL applies it, so the bootstrapper never re-reads to verify
/// its own ALTER. A boot whose TTL already matches issues no ALTER.</item>
/// <item><b>Failure:</b> not latched. Within <see cref="BootstrapFailureCache.DefaultDuration"/> the last failure is
/// re-thrown without touching QuestDB.</item>
/// <item><c>AutoCreateSchema=false</c> only verifies the tables exist (no TTL changes) and throws
/// <see cref="HistorianSchemaMissingException"/> otherwise.</item>
/// </list>
/// </summary>
internal sealed class QuestDbSchemaBootstrapper
{
private readonly QuestDbConnection _conn;
private readonly QuestDbHistorianOptions _options;
private readonly ILogger _log;
private readonly SemaphoreSlim _gate = new(1, 1);
private readonly BootstrapFailureCache _failure; // guarded by _gate
private volatile bool _ready;
/// <summary>Creates the bootstrapper.</summary>
/// <param name="conn">Connection helper.</param>
/// <param name="options">Validated options.</param>
/// <param name="log">Optional logger.</param>
/// <param name="timeProvider">Clock for the failure cache; system by default.</param>
public QuestDbSchemaBootstrapper(QuestDbConnection conn, QuestDbHistorianOptions options, ILogger? log = null,
TimeProvider? timeProvider = null)
{
_conn = conn;
_options = options;
_log = log ?? NullLogger.Instance;
_failure = new BootstrapFailureCache(timeProvider ?? TimeProvider.System, BootstrapFailureCache.DefaultDuration);
}
/// <summary>Ensures the schema is ready; no-op after the first success.</summary>
/// <param name="ct">Cancellation.</param>
/// <returns>A task completing when the tables are usable.</returns>
/// <exception cref="HistorianSchemaMissingException">AutoCreateSchema is false and a table is absent.</exception>
public async Task EnsureAsync(CancellationToken ct)
{
if (_ready) return;
await _gate.WaitAsync(ct).ConfigureAwait(false);
try
{
if (_ready) return;
_failure.ThrowIfCached();
try
{
await using var conn = await _conn.OpenAsync(ct).ConfigureAwait(false);
if (_options.AutoCreateSchema)
{
await CreateTablesAsync(conn, ct).ConfigureAwait(false);
await ReconcileTtlAsync(conn, ct).ConfigureAwait(false);
}
else
{
await VerifyAsync(conn, ct).ConfigureAwait(false);
}
_ready = true;
_failure.Clear();
}
catch (Exception ex) when (ex is not OperationCanceledException || !ct.IsCancellationRequested)
{
_failure.Record(ex);
_log.LogWarning("QuestDB historian schema bootstrap (prefix '{Prefix}') failed ({Exception}: {Message}); " +
"historian calls fail fast for {Seconds} s before retrying.",
_conn.TablePrefix, ex.GetType().Name, ex.Message, BootstrapFailureCache.DefaultDuration.TotalSeconds);
throw;
}
}
finally
{
_gate.Release();
}
}
private async Task VerifyAsync(NpgsqlConnection conn, CancellationToken ct)
{
foreach (var table in QuestDbTables.All(_conn.TablePrefix))
{
if (!await _conn.TableExistsAsync(conn, table, ct).ConfigureAwait(false))
throw new HistorianSchemaMissingException(
$"QuestDB historian table '{table}' is missing and AutoCreateSchema is false. " +
"Create the tables (see docs/Historian.md) or enable AutoCreateSchema.");
}
}
private async Task CreateTablesAsync(NpgsqlConnection conn, CancellationToken ct)
{
var statements = QuestDbTables.CreateAll(_conn.TablePrefix, _options.RetentionDays);
var tables = QuestDbTables.All(_conn.TablePrefix);
for (var i = 0; i < statements.Length; i++)
{
try
{
await using var cmd = _conn.CreateCommand(conn, statements[i]);
await cmd.ExecuteNonQueryAsync(ct).ConfigureAwait(false);
}
catch (PostgresException ex) when (QuestDbConnection.IsAlreadyExists(ex))
{
// A concurrent creator won the race - swallow ONLY if the table is really there.
if (!await _conn.TableExistsAsync(conn, tables[i], ct).ConfigureAwait(false)) throw;
_log.LogDebug("QuestDB table {Table} was created concurrently.", tables[i]);
}
}
}
private async Task ReconcileTtlAsync(NpgsqlConnection conn, CancellationToken ct)
{
var wantHours = _options.RetentionDays is { } d ? d * 24L : 0L;
foreach (var table in QuestDbTables.Expiring(_conn.TablePrefix))
{
long haveHours;
await using (var cmd = _conn.CreateCommand(conn, "SELECT ttlValue, ttlUnit FROM tables() WHERE table_name = $1"))
{
QuestDbConnection.BindText(cmd, table);
await using var r = await cmd.ExecuteReaderAsync(ct).ConfigureAwait(false);
if (!await r.ReadAsync(ct).ConfigureAwait(false)) continue; // vanished (dropped between create and here)
var value = r.IsDBNull(0) ? 0L : Convert.ToInt64(r.GetValue(0));
var unit = r.IsDBNull(1) ? null : r.GetString(1);
haveHours = QuestDbTables.TtlHours(value, unit);
}
if (haveHours == wantHours) continue;
// The ALTER is a WAL transaction: tables() keeps the old value until it applies, so do NOT re-read to verify.
await using (var alter = _conn.CreateCommand(conn, QuestDbTables.AlterTtl(table, _options.RetentionDays)))
await alter.ExecuteNonQueryAsync(ct).ConfigureAwait(false);
_log.LogInformation("Applied TTL {Days} day(s) to QuestDB table {Table}.", _options.RetentionDays, table);
}
}
}
src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbTables.cs
+83
View File
@@ -0,0 +1,83 @@
using System.Globalization;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb;
/// <summary>
/// Table names and DDL for the QuestDB backend (T0-confirmed against QuestDB 10.0.1). The TTL clause sits
/// between <c>PARTITION BY</c> and <c>WAL</c> and is an inlined integer (QuestDB cannot parameterise it).
/// Prefixes are regex-validated by the caller; these builders also refuse an invalid one.
/// </summary>
internal static class QuestDbTables
{
/// <summary>Base name of the sample table (prefix + this).</summary>
public const string SampleBase = "sample";
/// <summary>Base name of the event table.</summary>
public const string EventBase = "event";
/// <summary>Base name of the tag catalog table.</summary>
public const string TagBase = "tag";
/// <summary>The fixed designated timestamp of every <c>tag</c> row, which makes the dedup key an upsert by name.</summary>
public static readonly DateTime TagEpoch = DateTime.UnixEpoch;
/// <summary>The prefixed sample table name.</summary>
public static string Sample(string prefix) => Name(prefix, SampleBase);
/// <summary>The prefixed event table name.</summary>
public static string Event(string prefix) => Name(prefix, EventBase);
/// <summary>The prefixed tag table name.</summary>
public static string Tag(string prefix) => Name(prefix, TagBase);
/// <summary>All three table names in creation order.</summary>
public static string[] All(string prefix) => [Sample(prefix), Event(prefix), Tag(prefix)];
/// <summary>The tables that carry a TTL (sample and event; the tag catalog never expires).</summary>
public static string[] Expiring(string prefix) => [Sample(prefix), Event(prefix)];
/// <summary><c>CREATE TABLE IF NOT EXISTS</c> for the sample table.</summary>
public static string CreateSample(string prefix, int? retentionDays) =>
$"CREATE TABLE IF NOT EXISTS {Sample(prefix)} (ts TIMESTAMP, tag SYMBOL, kind BYTE, num_value DOUBLE, " +
"int_value LONG, str_value VARCHAR, status LONG, deleted BOOLEAN, server_ts TIMESTAMP) " +
$"TIMESTAMP(ts) PARTITION BY DAY{Ttl(retentionDays)} WAL DEDUP UPSERT KEYS(ts, tag)";
/// <summary><c>CREATE TABLE IF NOT EXISTS</c> for the event table.</summary>
public static string CreateEvent(string prefix, int? retentionDays) =>
$"CREATE TABLE IF NOT EXISTS {Event(prefix)} (ts TIMESTAMP, event_id VARCHAR, source SYMBOL, alarm_id VARCHAR, " +
"alarm_name VARCHAR, alarm_type_name VARCHAR, severity INT, event_kind VARCHAR, message VARCHAR, " +
"user_name VARCHAR, comment VARCHAR, received_utc TIMESTAMP) " +
$"TIMESTAMP(ts) PARTITION BY DAY{Ttl(retentionDays)} WAL DEDUP UPSERT KEYS(ts, event_id)";
/// <summary><c>CREATE TABLE IF NOT EXISTS</c> for the tag catalog (YEAR partitions, no TTL ever).</summary>
public static string CreateTag(string prefix) =>
$"CREATE TABLE IF NOT EXISTS {Tag(prefix)} (ts TIMESTAMP, name SYMBOL, data_type BYTE, eng_unit VARCHAR, " +
"description VARCHAR, updated_utc TIMESTAMP) TIMESTAMP(ts) PARTITION BY YEAR WAL DEDUP UPSERT KEYS(ts, name)";
/// <summary>The three CREATE statements in creation order.</summary>
public static string[] CreateAll(string prefix, int? retentionDays) =>
[CreateSample(prefix, retentionDays), CreateEvent(prefix, retentionDays), CreateTag(prefix)];
/// <summary><c>ALTER TABLE t SET TTL n DAYS</c>; <c>null</c> removes the TTL (<c>SET TTL 0 DAYS</c>).</summary>
public static string AlterTtl(string table, int? retentionDays) =>
$"ALTER TABLE {table} SET TTL {(retentionDays ?? 0).ToString(CultureInfo.InvariantCulture)} DAYS";
/// <summary>
/// Converts a <c>tables()</c> <c>ttlValue</c>/<c>ttlUnit</c> pair to hours. QuestDB normalises the unit (7 days
/// reads back as <c>1 WEEK</c>), so equality is judged in hours. An unrecognised unit returns -1 (never equal to
/// a wanted value, so the bootstrapper re-applies the TTL).
/// </summary>
public static long TtlHours(long value, string? unit) => (unit ?? "").ToUpperInvariant() switch
{
"HOUR" => value,
"DAY" => value * 24,
"WEEK" => value * 168,
_ => value == 0 ? 0 : -1,
};
private static string Ttl(int? days) =>
days is { } d ? $" TTL {d.ToString(CultureInfo.InvariantCulture)} DAYS" : "";
private static string Name(string prefix, string baseName)
{
if (!QuestDbHistorianOptions.IsValidPrefix(prefix))
throw new ArgumentException("Invalid table prefix.", nameof(prefix));
return prefix + baseName;
}
}
src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbTagCatalog.cs
+101
View File
@@ -0,0 +1,101 @@
using System.Collections.Concurrent;
using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb;
/// <summary>
/// The only "is this tag known" authority for the QuestDB backend (R7): a row in the tag table (fixed designated
/// timestamp 1970-01-01, deduplicated by <c>name</c>). Caches positive results only. Both methods run the schema
/// bootstrap first. <b>Names longer than <see cref="HistorianTagNames.MaxLength"/> are never truncated</b>:
/// <see cref="ExistsAsync"/> answers false and <see cref="EnsureAsync"/> throws.
/// <para><see cref="EnsureAsync"/> inserts only when the tag is absent, because the table's dedup key replaces a
/// whole row: re-inserting a known tag would erase the engineering unit and description the provisioner wrote.</para>
/// </summary>
internal sealed class QuestDbTagCatalog
{
private readonly QuestDbConnection _conn;
private readonly QuestDbSchemaBootstrapper _bootstrap;
private readonly QuestDbWalVisibility _wal;
private readonly TimeProvider _clock;
private readonly ConcurrentDictionary<string, byte> _known = new(StringComparer.Ordinal);
/// <summary>Creates the catalog.</summary>
/// <param name="conn">Connection helper.</param>
/// <param name="bootstrap">Schema bootstrapper.</param>
/// <param name="wal">WAL visibility wait.</param>
/// <param name="clock">Clock for <c>updated_utc</c>; system by default.</param>
public QuestDbTagCatalog(QuestDbConnection conn, QuestDbSchemaBootstrapper bootstrap, QuestDbWalVisibility wal,
TimeProvider? clock = null)
{
_conn = conn;
_bootstrap = bootstrap;
_wal = wal;
_clock = clock ?? TimeProvider.System;
}
/// <summary>Read-only lookup; never creates a row.</summary>
/// <param name="name">Historian tag name.</param>
/// <param name="ct">Cancellation.</param>
/// <returns>True when the tag is in the catalog; false for an unknown or over-long name.</returns>
public async Task<bool> ExistsAsync(string name, CancellationToken ct)
{
if (!HistorianTagNames.IsValid(name)) return false;
if (_known.ContainsKey(name)) return true;
await _bootstrap.EnsureAsync(ct).ConfigureAwait(false);
var found = await StoredAsync(name, ct).ConfigureAwait(false);
if (found) _known[name] = 0;
return found;
}
private async Task<bool> StoredAsync(string name, CancellationToken ct)
{
await using var conn = await _conn.OpenAsync(ct).ConfigureAwait(false);
await using var cmd = _conn.CreateCommand(conn, $"SELECT count() FROM {_conn.TagTable} WHERE name = $1");
QuestDbConnection.BindText(cmd, name);
return Convert.ToInt64(await cmd.ExecuteScalarAsync(ct).ConfigureAwait(false)) > 0;
}
/// <summary>
/// Registers the tag when absent (no-op when present) and waits for the row to become visible. On a miss it first
/// waits for the tag table's already-sequenced commits and re-checks, so a row the provisioner has committed but
/// QuestDB has not yet applied is found rather than overwritten (its unit/description kept).
/// </summary>
/// <param name="name">Historian tag name.</param>
/// <param name="dataType">Initial <c>data_type</c> (only written on create; provisioning corrects it later).</param>
/// <param name="ct">Cancellation.</param>
/// <returns>A task completing when the tag is registered and visible.</returns>
/// <exception cref="ArgumentException">The name is longer than <see cref="HistorianTagNames.MaxLength"/>.</exception>
public async Task EnsureAsync(string name, byte dataType, CancellationToken ct)
{
if (!HistorianTagNames.IsValid(name))
throw new ArgumentException($"Historian tag name exceeds {HistorianTagNames.MaxLength} characters.", nameof(name));
if (await ExistsAsync(name, ct).ConfigureAwait(false)) return;
// The miss may only mean another writer's row (the provisioner's, carrying eng_unit/description) is committed
// but not yet applied. Inserting now would land AFTER it and the dedup key would replace it with a row that has
// no metadata, so first wait for everything already sequenced on the tag table and look again.
await _wal.WaitAsync(_conn.TagTable, ct).ConfigureAwait(false);
if (await StoredAsync(name, ct).ConfigureAwait(false))
{
_known[name] = 0;
return;
}
await using (var conn = await _conn.OpenAsync(ct).ConfigureAwait(false))
{
await using var cmd = _conn.CreateCommand(conn,
$"INSERT INTO {_conn.TagTable} (ts, name, data_type, updated_utc) VALUES ($1, $2, $3, $4)");
QuestDbConnection.BindTimestamp(cmd, QuestDbTables.TagEpoch);
QuestDbConnection.BindText(cmd, name);
QuestDbConnection.BindByte(cmd, dataType);
QuestDbConnection.BindTimestamp(cmd, _clock.GetUtcNow().UtcDateTime);
await cmd.ExecuteNonQueryAsync(ct).ConfigureAwait(false);
}
await _wal.WaitAsync(_conn.TagTable, ct).ConfigureAwait(false);
_known[name] = 0;
}
/// <summary>Marks a tag as known after another component (the provisioner) has written its row and waited.</summary>
/// <param name="name">Historian tag name.</param>
internal void MarkKnown(string name) => _known[name] = 0;
}
src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbTagProvisioner.cs
+141
View File
@@ -0,0 +1,141 @@
using System.Text;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Logging.Abstractions;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb;
/// <summary>
/// <see cref="IHistorianProvisioning"/> for the QuestDB backend: upserts <c>tag</c> rows (fixed designated timestamp
/// 1970-01-01, so the dedup key makes it an upsert by name). QuestDB replaces the WHOLE row on a key hit, so a null
/// engineering unit / description in a request would wipe the stored one; per chunk the existing rows are read
/// (<c>WHERE name IN (…)</c>) and <c>COALESCE(request, stored)</c> is applied client-side before inserting.
/// Every data type is provisionable (Skipped is always 0). Never throws. After each chunk it waits for WAL
/// visibility and tells the catalog the names are known (R7).
/// </summary>
internal sealed class QuestDbTagProvisioner : IHistorianProvisioning
{
private const int ChunkSize = 400; // 6 parameters per tag on insert, 400 on the IN list
private const int MaxUnitLength = 64;
private const int MaxDescriptionLength = 400;
private readonly QuestDbConnection _conn;
private readonly QuestDbSchemaBootstrapper _bootstrap;
private readonly QuestDbTagCatalog _catalog;
private readonly QuestDbWalVisibility _wal;
private readonly ILogger _log;
private readonly TimeProvider _clock;
/// <summary>Creates the provisioner.</summary>
/// <param name="conn">Connection helper.</param>
/// <param name="bootstrap">Schema bootstrapper.</param>
/// <param name="catalog">Tag catalog (told which names are now known).</param>
/// <param name="wal">WAL visibility wait.</param>
/// <param name="log">Optional logger.</param>
/// <param name="clock">Clock for <c>updated_utc</c>.</param>
public QuestDbTagProvisioner(QuestDbConnection conn, QuestDbSchemaBootstrapper bootstrap, QuestDbTagCatalog catalog,
QuestDbWalVisibility wal, ILogger? log = null, TimeProvider? clock = null)
{
_conn = conn;
_bootstrap = bootstrap;
_catalog = catalog;
_wal = wal;
_log = log ?? NullLogger.Instance;
_clock = clock ?? TimeProvider.System;
}
/// <inheritdoc/>
public async Task<HistorianProvisionResult> EnsureTagsAsync(IReadOnlyList<HistorianTagProvisionRequest> requests, CancellationToken ct)
{
var requested = requests?.Count ?? 0;
if (requested == 0) return new HistorianProvisionResult(0, 0, 0, 0);
var failed = 0;
var ensured = 0;
try
{
await _bootstrap.EnsureAsync(ct).ConfigureAwait(false);
// Distinct names (last request wins); over-long/empty names fail alone. Weight = requests folded into the name.
var byName = new Dictionary<string, (HistorianTagProvisionRequest Req, int Count)>(StringComparer.Ordinal);
foreach (var r in requests!)
{
if (r is null || string.IsNullOrWhiteSpace(r.TagName) || !HistorianTagNames.IsValid(r.TagName)) { failed++; continue; }
byName[r.TagName] = (r, byName.TryGetValue(r.TagName, out var e) ? e.Count + 1 : 1);
}
var ordered = byName.OrderBy(kv => kv.Key, StringComparer.Ordinal).Select(kv => kv.Value).ToList();
for (var i = 0; i < ordered.Count; i += ChunkSize)
{
var chunk = ordered.Skip(i).Take(ChunkSize).ToList();
var weight = chunk.Sum(c => c.Count);
if (await TryUpsertAsync(chunk.Select(c => c.Req).ToList(), ct).ConfigureAwait(false)) ensured += weight;
else failed += weight;
}
}
catch (Exception ex)
{
_log.LogWarning("QuestDB historian provisioning failed ({Exception}).", ex.GetType().Name);
failed = requested - ensured;
}
return new HistorianProvisionResult(requested, ensured, 0, failed);
}
private async Task<bool> TryUpsertAsync(List<HistorianTagProvisionRequest> chunk, CancellationToken ct)
{
try
{
var stored = new Dictionary<string, (string? Unit, string? Desc)>(StringComparer.Ordinal);
await using (var conn = await _conn.OpenAsync(ct).ConfigureAwait(false))
{
var sel = new StringBuilder($"SELECT name, eng_unit, description FROM {_conn.TagTable} WHERE name IN (");
await using var cmd = _conn.CreateCommand(conn, "");
for (var i = 0; i < chunk.Count; i++)
{
sel.Append(i == 0 ? "$" : ",$").Append(i + 1);
QuestDbConnection.BindText(cmd, chunk[i].TagName);
}
cmd.CommandText = sel.Append(')').ToString();
await using var r = await cmd.ExecuteReaderAsync(ct).ConfigureAwait(false);
while (await r.ReadAsync(ct).ConfigureAwait(false))
stored[r.GetString(0)] = (r.IsDBNull(1) ? null : r.GetString(1), r.IsDBNull(2) ? null : r.GetString(2));
}
var now = _clock.GetUtcNow().UtcDateTime;
await using (var conn = await _conn.OpenAsync(ct).ConfigureAwait(false))
{
var ins = new StringBuilder($"INSERT INTO {_conn.TagTable} (ts, name, data_type, eng_unit, description, updated_utc) VALUES ");
await using var cmd = _conn.CreateCommand(conn, "");
var n = 1;
for (var i = 0; i < chunk.Count; i++)
{
var req = chunk[i];
stored.TryGetValue(req.TagName, out var old);
if (i > 0) ins.Append(',');
ins.Append($"(${n},${n + 1},${n + 2},${n + 3},${n + 4},${n + 5})");
n += 6;
QuestDbConnection.BindTimestamp(cmd, QuestDbTables.TagEpoch);
QuestDbConnection.BindText(cmd, req.TagName);
QuestDbConnection.BindByte(cmd, (byte)req.DataType);
QuestDbConnection.BindText(cmd, Clip(req.EngineeringUnit ?? old.Unit, MaxUnitLength));
QuestDbConnection.BindText(cmd, Clip(req.Description ?? old.Desc, MaxDescriptionLength));
QuestDbConnection.BindTimestamp(cmd, now);
}
cmd.CommandText = ins.ToString();
await cmd.ExecuteNonQueryAsync(ct).ConfigureAwait(false);
}
await _wal.WaitAsync(_conn.TagTable, ct).ConfigureAwait(false);
foreach (var c in chunk) _catalog.MarkKnown(c.TagName);
return true;
}
catch (Exception ex)
{
_log.LogWarning("QuestDB historian tag upsert failed ({Exception}).", ex.GetType().Name);
return false;
}
}
private static string? Clip(string? s, int max) => s is null ? null : s.Length <= max ? s : s[..max];
}
src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbTimeClusters.cs
+49
View File
@@ -0,0 +1,49 @@
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb;
/// <summary>
/// Groups the times of one write or HistoryUpdate request into inclusive <c>[Lo, Hi]</c> ranges so that the
/// "which of these times already exist" check reads one range query per cluster instead of one query spanning
/// <c>[min, max]</c> of the whole request. One far-away outlier time then costs its own tiny query instead of
/// widening a single range over every stored row in between. Shared by the live-value writer and the updater.
/// </summary>
internal static class QuestDbTimeClusters
{
/// <summary>Request times further apart than this start a new existence-query range.</summary>
internal static readonly TimeSpan ExistenceGap = TimeSpan.FromMinutes(1);
/// <summary>
/// Splits <paramref name="times"/> (any order; sorted here) into inclusive ranges, ascending: a new range starts
/// where the gap to the previous time exceeds <paramref name="gap"/> (exactly <paramref name="gap"/> stays
/// together), or where the current range already holds <paramref name="maxPerCluster"/> times.
/// </summary>
/// <param name="times">Request times, in any order.</param>
/// <param name="gap">Largest gap kept inside one range.</param>
/// <param name="maxPerCluster">Most times in one range (&gt;= 1); unbounded by default.</param>
/// <returns>The ranges, ascending; empty for no times.</returns>
internal static IReadOnlyList<(DateTime Lo, DateTime Hi)> Cluster(
IReadOnlyList<DateTime> times, TimeSpan gap, int maxPerCluster = int.MaxValue)
{
ArgumentNullException.ThrowIfNull(times);
ArgumentOutOfRangeException.ThrowIfLessThan(maxPerCluster, 1);
var result = new List<(DateTime, DateTime)>();
if (times.Count == 0) return result;
var sorted = times.Order().ToList();
var lo = sorted[0];
var prev = lo;
var n = 1;
for (var i = 1; i < sorted.Count; i++)
{
var t = sorted[i];
if (t - prev > gap || n >= maxPerCluster)
{
result.Add((lo, prev));
lo = t;
n = 0;
}
prev = t;
n++;
}
result.Add((lo, prev));
return result;
}
}
src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbWalVisibility.cs
+92
View File
@@ -0,0 +1,92 @@
using System.Diagnostics;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Logging.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb;
/// <summary>
/// The read-your-writes wait (R5, T0 Q4). QuestDB applies WAL commits asynchronously, so a row just inserted may
/// not be visible to the next query. After its INSERT(s) a writer calls <see cref="WaitAsync"/>, which captures
/// <c>sequencerTxn</c> from <c>wal_tables()</c> (every autocommitted INSERT is one sequencer transaction; a concurrent
/// writer can only raise the target) and polls <c>writerTxn</c> every ~10 ms until it reaches the target, the table
/// is suspended, or <see cref="QuestDbHistorianOptions.VisibilityTimeout"/> elapses.
/// </summary>
internal sealed class QuestDbWalVisibility
{
private static readonly TimeSpan PollInterval = TimeSpan.FromMilliseconds(10);
private readonly QuestDbConnection _conn;
private readonly TimeSpan _timeout;
private readonly ILogger _log;
/// <summary>Creates the waiter.</summary>
/// <param name="conn">Connection helper.</param>
/// <param name="timeout">Maximum wait; <see cref="TimeSpan.Zero"/> (or less) disables the wait.</param>
/// <param name="log">Optional logger.</param>
public QuestDbWalVisibility(QuestDbConnection conn, TimeSpan timeout, ILogger? log = null)
{
_conn = conn;
_timeout = timeout;
_log = log ?? NullLogger.Instance;
}
/// <summary>True when the wait is disabled (<c>VisibilityTimeout</c> is 0).</summary>
public bool Disabled => _timeout <= TimeSpan.Zero;
/// <summary>True when the applied transaction has caught up with the target, or the table cannot progress.</summary>
/// <param name="writerTxn">Last applied transaction.</param>
/// <param name="targetTxn">The sequencer transaction captured after the INSERT.</param>
/// <returns>Whether polling can stop because the rows are visible.</returns>
internal static bool Reached(long writerTxn, long targetTxn) => writerTxn >= targetTxn;
/// <summary>Waits until everything this caller inserted into <paramref name="table"/> is visible to queries.</summary>
/// <param name="table">Full table name written to.</param>
/// <param name="ct">Cancellation.</param>
/// <returns>
/// True when visible (or the wait is disabled, or the table is not a WAL table); false when the table is suspended
/// or the timeout elapsed (a Warning is logged; the rows are still committed and become visible later).
/// </returns>
public async Task<bool> WaitAsync(string table, CancellationToken ct)
{
if (Disabled) return true;
var sw = Stopwatch.StartNew();
await using var conn = await _conn.OpenAsync(ct).ConfigureAwait(false);
long target;
await using (var cmd = _conn.CreateCommand(conn, "SELECT sequencerTxn FROM wal_tables() WHERE name = $1"))
{
QuestDbConnection.BindText(cmd, table);
var o = await cmd.ExecuteScalarAsync(ct).ConfigureAwait(false);
if (o is null or DBNull) return true; // not a WAL table: writes are immediately visible
target = Convert.ToInt64(o);
}
while (true)
{
long writer;
bool suspended;
await using (var cmd = _conn.CreateCommand(conn, "SELECT writerTxn, suspended FROM wal_tables() WHERE name = $1"))
{
QuestDbConnection.BindText(cmd, table);
await using var r = await cmd.ExecuteReaderAsync(ct).ConfigureAwait(false);
if (!await r.ReadAsync(ct).ConfigureAwait(false)) return true;
writer = r.GetInt64(0);
suspended = r.GetBoolean(1);
}
if (Reached(writer, target)) return true;
if (suspended)
{
_log.LogWarning("QuestDB table {Table} is suspended (writerTxn {Writer} < sequencerTxn {Target}); not waiting.",
table, writer, target);
return false;
}
if (sw.Elapsed >= _timeout)
{
_log.LogWarning("QuestDB table {Table} did not become visible within {Timeout} ms (writerTxn {Writer} < sequencerTxn {Target}).",
table, _timeout.TotalMilliseconds, writer, target);
return false;
}
await Task.Delay(PollInterval, ct).ConfigureAwait(false);
}
}
}
src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.csproj
+30
View File
@@ -0,0 +1,30 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
<GenerateDocumentationFile>true</GenerateDocumentationFile>
<NoWarn>$(NoWarn);CS1591</NoWarn>
<Platforms>AnyCPU;x64</Platforms>
<RootNamespace>ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb</RootNamespace>
</PropertyGroup>
<ItemGroup>
<ProjectReference Include="..\..\Core\ZB.MOM.WW.OtOpcUa.Core.Abstractions\ZB.MOM.WW.OtOpcUa.Core.Abstractions.csproj"/>
<ProjectReference Include="..\..\Core\ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian\ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian.csproj"/>
<!-- The database-independent historian core shared with the SQL Server backend. -->
<ProjectReference Include="..\ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common\ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.csproj"/>
<!-- Runtime owns IHistorianBackendProvider / ServerHistorianOptions (same diamond as the SqlServer backend). -->
<ProjectReference Include="..\..\Server\ZB.MOM.WW.OtOpcUa.Runtime\ZB.MOM.WW.OtOpcUa.Runtime.csproj"/>
</ItemGroup>
<ItemGroup>
<PackageReference Include="Npgsql" />
<PackageReference Include="Microsoft.Extensions.Logging.Abstractions" />
</ItemGroup>
<ItemGroup>
<InternalsVisibleTo Include="ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests"/>
<InternalsVisibleTo Include="ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests"/>
</ItemGroup>
</Project>
src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/AggregateEvaluator.cs
+78
View File
@@ -0,0 +1,78 @@
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common;
/// <summary>
/// Maps one bucket's statistics (<see cref="BucketRow"/>) to the OPC UA value and status of an aggregate.
/// </summary>
/// <remarks>
/// <para>
/// Per interval: no samples ⇒ <c>BadNoData</c>. PercentGood / PercentBad ⇒ <c>100 × good|bad / all</c> over
/// every sample, status Good. Otherwise no usable sample ⇒ <c>BadNoData</c>; a numeric aggregate with a usable
/// String/DateTime sample ⇒ <c>BadAggregateInvalidInputs</c>; else the value, Good when every sample is Good,
/// <c>UncertainDataSubNormal</c> when any is Bad or Uncertain. Every status — including <c>BadNoData</c> —
/// carries the Calculated info bits. Timestamps are the bucket start, except Minimum/MaximumActualTime
/// (the sample's time; ties ⇒ earliest).
/// </para>
/// <para>
/// Values: Count is <see cref="int"/>; Start/End keep the sample's type; every other aggregate is
/// <see cref="double"/> computed over the numeric value (Boolean = 0/1). These are the simple per-sample
/// definitions, not Part 13's interval-boundary (interpolated bound / time-weighted) semantics.
/// </para>
/// </remarks>
public static class AggregateEvaluator
{
/// <summary>Evaluates <paramref name="aggregate"/> over one non-empty bucket.</summary>
/// <param name="aggregate">Requested aggregate.</param>
/// <param name="r">The bucket's statistics.</param>
/// <param name="bucketStart">The bucket's start time (the result timestamp).</param>
/// <param name="now">Server timestamp.</param>
/// <returns>The calculated value.</returns>
/// <exception cref="HistoryAggregateNotSupportedException">The aggregate is not one of the thirteen served.</exception>
public static DataValueSnapshot Evaluate(HistoryAggregateType aggregate, BucketRow r, DateTime bucketStart, DateTime now)
{
DataValueSnapshot Calc(object? value, uint status, DateTime? t = null) =>
new(value, HistorianInfoBits.WithCalculated(status), t ?? bucketStart, now);
if (r.Total == 0) return NoData(bucketStart, now);
switch (aggregate)
{
case HistoryAggregateType.PercentGood: return Calc(100d * r.Good / r.Total, HistorianStatusCodes.Good);
case HistoryAggregateType.PercentBad: return Calc(100d * r.Bad / r.Total, HistorianStatusCodes.Good);
}
if (r.Usable == 0) return NoData(bucketStart, now);
var anyText = aggregate is HistoryAggregateType.Count or HistoryAggregateType.Start or HistoryAggregateType.End;
if (!anyText && r.NonNumeric > 0)
return Calc(null, HistorianStatusCodes.BadAggregateInvalidInputs);
var status = r.Good == r.Total ? HistorianStatusCodes.Good : HistorianStatusCodes.UncertainDataSubNormal;
switch (aggregate)
{
case HistoryAggregateType.Count: return Calc(r.Usable, status);
case HistoryAggregateType.Average: return Calc(r.Avg, status);
case HistoryAggregateType.Minimum: return Calc(r.Min, status);
case HistoryAggregateType.Maximum: return Calc(r.Max, status);
case HistoryAggregateType.Total: return Calc(r.Sum, status);
case HistoryAggregateType.Range: return Calc(r.Max - r.Min, status);
case HistoryAggregateType.Start: return Calc(r.First.ToObject(), status);
case HistoryAggregateType.End: return Calc(r.Last.ToObject(), status);
case HistoryAggregateType.Delta:
return r.First.TryGetDouble(out var a) && r.Last.TryGetDouble(out var b)
? Calc(b - a, status)
: Calc(null, HistorianStatusCodes.BadAggregateInvalidInputs);
case HistoryAggregateType.MinimumActualTime:
case HistoryAggregateType.MaximumActualTime:
return Calc(r.Extreme, status, r.ExtremeTime);
default:
throw new HistoryAggregateNotSupportedException(aggregate);
}
}
/// <summary>The result for a bucket with no samples (or no usable sample): <c>BadNoData</c> + Calculated.</summary>
/// <param name="t">The bucket's start time.</param>
/// <param name="now">Server timestamp.</param>
/// <returns>The no-data value.</returns>
public static DataValueSnapshot NoData(DateTime t, DateTime now) =>
new(null, HistorianInfoBits.WithCalculated(HistorianStatusCodes.BadNoData), t, now);
}
src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/AggregateFamily.cs
+33
View File
@@ -0,0 +1,33 @@
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common;
/// <summary>The query shape that serves an aggregate. One query per family.</summary>
public enum AggregateFamily
{
/// <summary>Average / Minimum / Maximum / Total / Count / Range / PercentGood / PercentBad: one GROUP BY.</summary>
Simple,
/// <summary>Start / End / Delta: first + last usable sample per bucket.</summary>
StartEnd,
/// <summary>MinimumActualTime / MaximumActualTime: extreme usable sample per bucket, ties ⇒ earliest.</summary>
ActualTime,
}
/// <summary>Which aggregates the SQL historian backends serve, and by which family.</summary>
public static class AggregateFamilies
{
/// <summary>The family that serves <paramref name="aggregate"/>, or null when the backends do not.</summary>
/// <param name="aggregate">Requested aggregate.</param>
/// <returns>The family, or null for an unsupported aggregate.</returns>
public static AggregateFamily? FamilyOf(HistoryAggregateType aggregate) => aggregate switch
{
HistoryAggregateType.Average or HistoryAggregateType.Minimum or HistoryAggregateType.Maximum
or HistoryAggregateType.Total or HistoryAggregateType.Count or HistoryAggregateType.Range
or HistoryAggregateType.PercentGood or HistoryAggregateType.PercentBad => AggregateFamily.Simple,
HistoryAggregateType.Start or HistoryAggregateType.End or HistoryAggregateType.Delta => AggregateFamily.StartEnd,
HistoryAggregateType.MinimumActualTime or HistoryAggregateType.MaximumActualTime => AggregateFamily.ActualTime,
_ => null,
};
}
src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/AlarmEventIds.cs
+61
View File
@@ -0,0 +1,61 @@
using System.Security.Cryptography;
using System.Text;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common;
/// <summary>
/// The deterministic alarm-event id and numeric severity the SQL historian backends store, so the store-and-forward
/// drain's at-least-once redelivery (and both pair nodes writing the same event) converge on one row.
/// </summary>
public static class AlarmEventIds
{
/// <summary>
/// Lowercase-hex SHA-256 over length-prefixed UTF-8 fields (little-endian int32 byte length, -1 for null):
/// EquipmentPath, AlarmId, EventKind, TimestampUtc.Ticks (decimal), User, Comment, Message. Length-prefixing keeps
/// field boundaries unambiguous (<c>"a|b"+"c"</c> ≠ <c>"a"+"b|c"</c>).
/// </summary>
/// <param name="e">The event.</param>
/// <returns>64 hex characters.</returns>
public static string ComputeEventId(AlarmHistorianEvent e)
{
using var ms = new MemoryStream();
Field(ms, e.EquipmentPath);
Field(ms, e.AlarmId);
Field(ms, e.EventKind);
Field(ms, e.TimestampUtc.Ticks.ToString(System.Globalization.CultureInfo.InvariantCulture));
Field(ms, e.User);
Field(ms, e.Comment);
Field(ms, e.Message);
return Convert.ToHexStringLower(SHA256.HashData(ms.ToArray()));
}
/// <summary>The numeric Part 9 priority stored for a severity (same values the gateway path sends).</summary>
/// <param name="s">Alarm severity.</param>
/// <returns>200 / 500 / 700 / 900; 500 for anything else.</returns>
public static int ToNumericSeverity(AlarmSeverity s) => s switch
{
AlarmSeverity.Low => 200,
AlarmSeverity.Medium => 500,
AlarmSeverity.High => 700,
AlarmSeverity.Critical => 900,
_ => 500,
};
private static void Field(MemoryStream ms, string? value)
{
Span<byte> len = stackalloc byte[4];
if (value is null)
{
System.Buffers.Binary.BinaryPrimitives.WriteInt32LittleEndian(len, -1);
ms.Write(len);
return;
}
var bytes = Encoding.UTF8.GetBytes(value);
System.Buffers.Binary.BinaryPrimitives.WriteInt32LittleEndian(len, bytes.Length);
ms.Write(len);
ms.Write(bytes);
}
}
src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/AtTimeEvaluator.cs
+67
View File
@@ -0,0 +1,67 @@
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common;
/// <summary>The AtTime rules, applied to one requested time and its stored neighbours.</summary>
public static class AtTimeEvaluator
{
/// <summary>
/// Applies the AtTime rules to one requested time <paramref name="t"/> (the result is stamped <c>t</c>):
/// <list type="bullet">
/// <item><c>prev</c> at exactly <c>t</c> ⇒ the stored value and status, unchanged.</item>
/// <item>No <c>prev</c> ⇒ <c>BadNoData</c>, null value.</item>
/// <item><c>prev</c> Bad (incl. an <c>Empty</c> outage marker) ⇒ <c>t</c> lies inside an outage: null value
/// carrying <c>prev</c>'s Bad status code (info bits stripped) — never better than its source.</item>
/// <item><c>prev</c> usable and <c>next</c> Bad ⇒ <c>prev</c>'s value held (stepped, NOT interpolated into the
/// outage) with <c>UncertainDataSubNormal</c> + Interpolated bits.</item>
/// <item>Double/Int64 with a usable <c>next</c> ⇒ linear interpolation (Int64 rounded to the nearest integer,
/// ties away from zero, keeping the type); status <c>Good</c> + Interpolated when both neighbours are Good,
/// else <c>UncertainDataSubNormal</c> + Interpolated.</item>
/// <item>Boolean/String/DateTime, or no <c>next</c> (after the last sample) ⇒ stepped: <c>prev</c>'s value with
/// <c>prev</c>'s status code + Interpolated bits — except that a time later than now that is stepped from the
/// last sample is <c>UncertainDataSubNormal</c> + Interpolated.</item>
/// </list>
/// </summary>
/// <param name="t">Requested time.</param>
/// <param name="prev">The stored sample at or before <paramref name="t"/>, if any.</param>
/// <param name="next">The stored sample after <paramref name="t"/>, if any.</param>
/// <param name="serverTime">Server timestamp for the result.</param>
/// <param name="now">The current time (for the beyond-now rule).</param>
/// <returns>The value at <paramref name="t"/>.</returns>
public static DataValueSnapshot Evaluate(DateTime t, StoredSample? prev, StoredSample? next, DateTime serverTime, DateTime now)
{
if (prev is not { } p)
return new DataValueSnapshot(null, HistorianStatusCodes.BadNoData, t, serverTime);
if (p.TimeUtc == t)
return new DataValueSnapshot(p.Value.ToObject(), p.Status, t, serverTime);
// Inside an outage: report the outage's own Bad code, never an upgraded Uncertain.
if (!p.IsUsable)
return new DataValueSnapshot(null, p.Status & StatusBits.CodeBitsMask, t, serverTime);
// Usable previous, Bad next: hold the previous value up to the outage; do not interpolate into it.
if (next is { IsUsable: false })
return new DataValueSnapshot(p.Value.ToObject(),
HistorianInfoBits.WithInterpolated(HistorianStatusCodes.UncertainDataSubNormal), t, serverTime);
if (next is { } n && p.Value.Kind is HistorianValueKind.Double or HistorianValueKind.Int64
&& n.Value.TryGetDouble(out var nv))
{
var fraction = (double)(t.Ticks - p.TimeUtc.Ticks) / (n.TimeUtc.Ticks - p.TimeUtc.Ticks);
var v = p.Value.Number + (nv - p.Value.Number) * fraction;
object value = p.Value.Kind == HistorianValueKind.Int64
? (long)Math.Round(v, MidpointRounding.AwayFromZero)
: v;
var status = p.IsGood && n.IsGood ? HistorianStatusCodes.Good : HistorianStatusCodes.UncertainDataSubNormal;
return new DataValueSnapshot(value, HistorianInfoBits.WithInterpolated(status), t, serverTime);
}
// Beyond the last sample AND beyond now: the value is held into the future, so it is not known to be Good.
if (next is null && t > now)
return new DataValueSnapshot(p.Value.ToObject(),
HistorianInfoBits.WithInterpolated(HistorianStatusCodes.UncertainDataSubNormal), t, serverTime);
// Stepped (non-numeric kind, or after the last sample): hold the previous value and its status.
return new DataValueSnapshot(p.Value.ToObject(), HistorianInfoBits.WithInterpolated(p.Status), t, serverTime);
}
}
src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/BootstrapFailureCache.cs
+46
View File
@@ -0,0 +1,46 @@
using System.Runtime.ExceptionServices;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common;
/// <summary>
/// The negative cache a schema bootstrapper keeps for its last failure: within the cache window the failure is
/// re-thrown without touching the database, so a missing schema (or an outage) does not serialise every write, read
/// and alarm event behind one semaphore and one round-trip each. A failure is NOT latched: after the window the
/// caller retries. Not thread-safe; the caller guards it (the bootstrapper's gate).
/// </summary>
public sealed class BootstrapFailureCache
{
/// <summary>How long the last bootstrap failure is re-thrown before the database is tried again (5 s).</summary>
public static readonly TimeSpan DefaultDuration = TimeSpan.FromSeconds(5);
private readonly TimeProvider _clock;
private readonly TimeSpan _duration;
private ExceptionDispatchInfo? _lastFailure;
private DateTimeOffset _retryAfter;
/// <summary>Creates the cache.</summary>
/// <param name="clock">Clock for the cache window.</param>
/// <param name="duration">How long a recorded failure is re-thrown (normally <see cref="DefaultDuration"/>).</param>
public BootstrapFailureCache(TimeProvider clock, TimeSpan duration)
{
_clock = clock ?? throw new ArgumentNullException(nameof(clock));
_duration = duration;
}
/// <summary>Re-throws the last recorded failure (original stack preserved) while its window is open.</summary>
public void ThrowIfCached()
{
if (_lastFailure is { } cached && _clock.GetUtcNow() < _retryAfter) cached.Throw();
}
/// <summary>Records a failure; it is re-thrown by <see cref="ThrowIfCached"/> for the cache window.</summary>
/// <param name="ex">The bootstrap failure.</param>
public void Record(Exception ex)
{
_lastFailure = ExceptionDispatchInfo.Capture(ex);
_retryAfter = _clock.GetUtcNow() + _duration;
}
/// <summary>Forgets any recorded failure (after a success).</summary>
public void Clear() => _lastFailure = null;
}
src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/BucketPlan.cs
+91
View File
@@ -0,0 +1,91 @@
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common;
/// <summary>
/// The bucket layout of a processed (aggregate) read. Bucket <c>i</c> = <c>[start + i·interval,
/// start + (i+1)·interval)</c> (start-inclusive, end-exclusive to the tick; the last bucket is truncated at
/// <c>end</c>). An interval &lt;= 0 ⇒ one interval <c>[start, end)</c>. <c>start == end</c> ⇒ no intervals.
/// <c>end &lt; start</c> (reverse) ⇒ the window is swapped, computed forward, and assembled in descending bucket
/// order (buckets stay anchored at the earlier time).
/// </summary>
public sealed class BucketPlan
{
/// <summary>Spec §9.7: a processed read covering more intervals than this is refused.</summary>
public const int MaxBuckets = 100_000;
private BucketPlan(DateTime startUtc, DateTime endUtc, bool reversed, long intervalTicks, long bucketCount)
{
StartUtc = startUtc;
EndUtc = endUtc;
Reversed = reversed;
IntervalTicks = intervalTicks;
BucketCount = bucketCount;
}
/// <summary>The earlier end of the window (the forward start).</summary>
public DateTime StartUtc { get; }
/// <summary>The later end of the window (the forward, exclusive end).</summary>
public DateTime EndUtc { get; }
/// <summary>True when the request came in reversed (<c>end &lt; start</c>).</summary>
public bool Reversed { get; }
/// <summary>Bucket width in 100 ns ticks (clamped to the window; 0 for an empty window).</summary>
public long IntervalTicks { get; }
/// <summary>Number of buckets (0 for an empty window).</summary>
public long BucketCount { get; }
/// <summary><see cref="StartUtc"/> as 100 ns ticks since the Unix epoch.</summary>
public long StartUnixTicks => (StartUtc - DateTime.UnixEpoch).Ticks;
/// <summary>Plans the buckets of a processed read.</summary>
/// <param name="startUtc">Requested start.</param>
/// <param name="endUtc">Requested end (earlier than start for a reverse read).</param>
/// <param name="interval">Bucket width; &lt;= 0 means one bucket over the whole window.</param>
/// <returns>The plan.</returns>
/// <exception cref="ArgumentOutOfRangeException">The window holds more than <see cref="MaxBuckets"/> intervals
/// (parameter name <c>interval</c>).</exception>
public static BucketPlan Create(DateTime startUtc, DateTime endUtc, TimeSpan interval)
{
var reverse = endUtc < startUtc;
var (start, end) = reverse ? (endUtc, startUtc) : (startUtc, endUtc);
var windowTicks = end.Ticks - start.Ticks;
// Everything stays in 100 ns ticks (≤ ≈3.2e18 across 0001–9999, inside a long) — no ns scaling that could
// wrap. An interval wider than the window is one bucket, so clamp it to the window; the ceiling division
// is written without "+ interval - 1" so a TimeSpan near MaxValue cannot overflow either.
var intervalTicks = interval > TimeSpan.Zero ? Math.Min(interval.Ticks, windowTicks) : windowTicks;
long bucketCount = windowTicks == 0 ? 0 : windowTicks / intervalTicks + (windowTicks % intervalTicks == 0 ? 0 : 1);
if (bucketCount > MaxBuckets)
throw new ArgumentOutOfRangeException(nameof(interval), bucketCount,
$"A processed read may cover at most {MaxBuckets} intervals; this window holds {bucketCount}.");
return new BucketPlan(start, end, reverse, intervalTicks, bucketCount);
}
/// <summary>The start time of bucket <paramref name="index"/> (UTC kind).</summary>
/// <param name="index">Zero-based bucket index.</param>
/// <returns>The bucket start.</returns>
public DateTime BucketStart(long index) =>
DateTime.SpecifyKind(new DateTime(StartUtc.Ticks + index * IntervalTicks), DateTimeKind.Utc);
/// <summary>One snapshot per bucket in window order; missing buckets get <paramref name="empty"/>; reversed if the window was.</summary>
/// <param name="byBucket">Evaluated snapshots of the non-empty buckets, by bucket index.</param>
/// <param name="empty">Builds the snapshot of a bucket with no row, from its start time.</param>
/// <returns><see cref="BucketCount"/> snapshots.</returns>
public IReadOnlyList<DataValueSnapshot> Assemble(IReadOnlyDictionary<long, DataValueSnapshot> byBucket,
Func<DateTime, DataValueSnapshot> empty)
{
ArgumentNullException.ThrowIfNull(byBucket);
ArgumentNullException.ThrowIfNull(empty);
var result = new List<DataValueSnapshot>((int)BucketCount);
for (long i = 0; i < BucketCount; i++)
{
var bucketStart = BucketStart(i);
result.Add(byBucket.TryGetValue(i, out var v) ? v : empty(bucketStart));
}
if (Reversed) result.Reverse();
return result;
}
}
src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/BucketRow.cs
+57
View File
@@ -0,0 +1,57 @@
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common;
/// <summary>
/// Per-bucket statistics one aggregate query row produces (one non-empty bucket). Every backend's SQL fills the same
/// fields; <see cref="AggregateEvaluator"/> turns them into the OPC UA value and status.
/// </summary>
/// <remarks>
/// Sample classes: <i>usable</i> = severity Good or Uncertain AND a stored value (an Empty sample is an outage marker
/// and counts as Bad whatever its code); <i>good</i> = usable with severity Good; <i>bad</i> = not usable.
/// Every value field is computed over usable samples only.
/// </remarks>
public sealed class BucketRow
{
/// <summary>Zero-based bucket index within the window.</summary>
public long Bucket { get; set; }
/// <summary>All samples in the bucket.</summary>
public int Total { get; set; }
/// <summary>Usable samples.</summary>
public int Usable { get; set; }
/// <summary>Good samples.</summary>
public int Good { get; set; }
/// <summary>Bad (not usable) samples.</summary>
public int Bad { get; set; }
/// <summary>Usable String / DateTime samples.</summary>
public int NonNumeric { get; set; }
/// <summary>Average of the usable numeric values (null when none).</summary>
public double? Avg { get; set; }
/// <summary>Minimum of the usable numeric values (null when none).</summary>
public double? Min { get; set; }
/// <summary>Maximum of the usable numeric values (null when none).</summary>
public double? Max { get; set; }
/// <summary>Sum of the usable numeric values (null when none).</summary>
public double? Sum { get; set; }
/// <summary>First usable sample's value by time (Empty when none).</summary>
public HistorianValue First { get; set; }
/// <summary>Last usable sample's value by time (Empty when none).</summary>
public HistorianValue Last { get; set; }
/// <summary>Extreme (min or max) usable numeric value (null when none).</summary>
public double? Extreme { get; set; }
/// <summary>Time of <see cref="Extreme"/> (ties ⇒ earliest; null when none).</summary>
public DateTime? ExtremeTime { get; set; }
}
src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/EventContinuation.cs
+11
View File
@@ -0,0 +1,11 @@
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common;
/// <summary>The opaque Events continuation point of the SQL historian backends (only its presence is contractual).</summary>
public static class EventContinuation
{
/// <summary>Encodes the first not-returned event as <c>"{EventTimeUtc.Ticks}|{EventId}"</c>.</summary>
/// <param name="eventTimeUtc">The event's time.</param>
/// <param name="eventId">The event's id.</param>
/// <returns>The continuation text (callers send its UTF-8 bytes).</returns>
public static string Encode(DateTime eventTimeUtc, string eventId) => $"{eventTimeUtc.Ticks}|{eventId}";
}
src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlHistorianDataTypes.cs → src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/HistorianDataTypes.cs
+4 -4
View File
@@ -1,18 +1,18 @@
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common;
/// <summary>
/// The one place that maps value kinds to <c>Tag.DataType</c> (the schema stores a <see cref="DriverDataType"/> byte).
/// The one place that maps value kinds to the stored tag data type (a <see cref="DriverDataType"/> byte).
/// Used by the value writer and the updater only when they have to create a tag row on a miss; provisioning
/// later corrects it.
/// </summary>
internal static class SqlHistorianDataTypes
public static class HistorianDataTypes
{
/// <summary>Data type used when a tag is created from a batch that holds no typed value.</summary>
public const DriverDataType DefaultForEmpty = DriverDataType.Float64;
/// <summary>Maps a value kind to the stored <c>Tag.DataType</c> byte (Empty ⇒ <see cref="DefaultForEmpty"/>).</summary>
/// <summary>Maps a value kind to the stored tag data-type byte (Empty ⇒ <see cref="DefaultForEmpty"/>).</summary>
/// <param name="kind">Value kind.</param>
/// <returns>The <see cref="DriverDataType"/> as a byte.</returns>
public static byte FromKind(HistorianValueKind kind) => (byte)(kind switch
src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/HistorianHealthCounters.cs
+70
View File
@@ -0,0 +1,70 @@
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common;
/// <summary>
/// The read-side health counters a SQL historian data source reports through
/// <see cref="IHistorianDataSource.GetHealthSnapshot"/>. Lock-guarded; safe to call from concurrent reads.
/// </summary>
public sealed class HistorianHealthCounters
{
private readonly object _lock = new();
private DateTime? _lastSuccessUtc;
private DateTime? _lastFailureUtc;
private string? _lastError;
private long _totalQueries;
private long _totalSuccesses;
private long _totalFailures;
private int _consecutiveFailures;
/// <summary>Records a successful query.</summary>
public void RecordSuccess()
{
lock (_lock)
{
_totalQueries++;
_totalSuccesses++;
_consecutiveFailures = 0;
_lastSuccessUtc = DateTime.UtcNow;
}
}
/// <summary>Records a failed query. An <see cref="OperationCanceledException"/> (deadline / client abort) is not a backend fault and is ignored.</summary>
/// <param name="ex">The failure.</param>
public void RecordFailure(Exception ex)
{
ArgumentNullException.ThrowIfNull(ex);
if (ex is OperationCanceledException) return;
lock (_lock)
{
_totalQueries++;
_totalFailures++;
_consecutiveFailures++;
_lastFailureUtc = DateTime.UtcNow;
_lastError = $"{ex.GetType().Name}: {ex.Message}";
}
}
/// <summary>The current counters. The connection is reported open when there has been a success and no failure since.</summary>
/// <returns>The snapshot.</returns>
public HistorianHealthSnapshot Snapshot()
{
lock (_lock)
{
var open = _totalSuccesses > 0 && _consecutiveFailures == 0;
return new HistorianHealthSnapshot(
TotalQueries: _totalQueries,
TotalSuccesses: _totalSuccesses,
TotalFailures: _totalFailures,
ConsecutiveFailures: _consecutiveFailures,
LastSuccessTime: _lastSuccessUtc,
LastFailureTime: _lastFailureUtc,
LastError: _lastError,
ProcessConnectionOpen: open,
EventConnectionOpen: open,
ActiveProcessNode: null,
ActiveEventNode: null,
Nodes: []);
}
}
}
src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/HistorianInfoBits.cs
+32
View File
@@ -0,0 +1,32 @@
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common;
/// <summary>Part 4 §7.39 historian info bits OR-ed onto a status code.</summary>
public static class HistorianInfoBits
{
/// <summary>InfoType = DataValue.</summary>
public const uint InfoTypeDataValue = 0x00000400u;
/// <summary>The value was produced by an aggregate calculation.</summary>
public const uint HistorianCalculated = 0x1u;
/// <summary>The value was interpolated.</summary>
public const uint HistorianInterpolated = 0x2u;
/// <summary>
/// Marks <paramref name="status"/> as a Calculated data value. Any historian bits the status already carries
/// (e.g. a stored Interpolated sample) are cleared first, so the result is never both.
/// </summary>
/// <param name="status">Base status code.</param>
/// <returns>Status with InfoType and Calculated bits set.</returns>
public static uint WithCalculated(uint status) =>
(status & ~StatusBits.HistorianBitsMask) | InfoTypeDataValue | HistorianCalculated;
/// <summary>
/// Marks <paramref name="status"/> as an Interpolated data value. Any historian bits the status already
/// carries (e.g. a stored Calculated sample) are cleared first, so the result is never both.
/// </summary>
/// <param name="status">Base status code.</param>
/// <returns>Status with InfoType and Interpolated bits set.</returns>
public static uint WithInterpolated(uint status) =>
(status & ~StatusBits.HistorianBitsMask) | InfoTypeDataValue | HistorianInterpolated;
}
src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/HistorianSampleRow.cs
+9
View File
@@ -0,0 +1,9 @@
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common;
/// <summary>One sample bound for storage.</summary>
/// <param name="TimeUtc">Source timestamp (UTC, at the backend's resolution).</param>
/// <param name="Value">Typed value.</param>
/// <param name="StatusCode">OPC UA status code (stored bit-for-bit).</param>
public readonly record struct HistorianSampleRow(DateTime TimeUtc, HistorianValue Value, uint StatusCode);
src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/HistorianSchemaMissingException.cs
+9
View File
@@ -0,0 +1,9 @@
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common;
/// <summary>Thrown when <c>AutoCreateSchema=false</c> and the historian schema is absent or behind.</summary>
public sealed class HistorianSchemaMissingException : Exception
{
/// <summary>Creates the exception.</summary>
/// <param name="message">Message naming the missing version.</param>
public HistorianSchemaMissingException(string message) : base(message) { }
}
src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/HistorianStatusCodes.cs
+27
View File
@@ -0,0 +1,27 @@
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common;
/// <summary>
/// Every OPC UA status the SQL historian backends emit, as named constants (inline literals are invisible to
/// <c>StatusCodeParityTests</c>). Values mirror <c>Opc.Ua.StatusCodes</c>.
/// </summary>
public static class HistorianStatusCodes
{
public const uint Good = 0x00000000u;
public const uint GoodEntryInserted = 0x00A20000u;
public const uint GoodEntryReplaced = 0x00A30000u;
public const uint BadNoData = 0x809B0000u;
public const uint BadEntryExists = 0x809F0000u;
public const uint BadNoEntryExists = 0x80A00000u;
public const uint BadInvalidTimestamp = 0x80230000u;
public const uint BadAggregateInvalidInputs = 0x80D60000u;
public const uint UncertainDataSubNormal = 0x40A40000u;
/// <summary>Stored for a non-finite Double (NaN / ±Infinity) that arrived with a non-Bad status.</summary>
public const uint BadOutOfRange = 0x803C0000u;
/// <summary>
/// A HistoryUpdate value that cannot be stored: a Double NaN / ±Infinity, an Empty (null) value with a non-Bad
/// status, or a tag name longer than <see cref="HistorianTagNames.MaxLength"/>.
/// </summary>
public const uint BadInvalidArgument = 0x80AB0000u;
}
src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/HistorianTagNames.cs
+17
View File
@@ -0,0 +1,17 @@
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common;
/// <summary>
/// Historian tag-name rules. <b>Names longer than <see cref="MaxLength"/></b> are never truncated: a sized parameter
/// would silently cut them and resolve them onto whatever tag owns the first <see cref="MaxLength"/> characters, so
/// they are refused (writes) or answered as not found (reads).
/// </summary>
public static class HistorianTagNames
{
/// <summary>Longest storable tag (and event source) name; longer names are refused.</summary>
public const int MaxLength = 400;
/// <summary>True when <paramref name="name"/> is storable without truncation.</summary>
/// <param name="name">Historian tag name.</param>
/// <returns>Whether the name is non-null and at most <see cref="MaxLength"/> characters.</returns>
public static bool IsValid(string? name) => name is not null && name.Length <= MaxLength;
}
src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/HistoricalEventRows.cs
+10
View File
@@ -0,0 +1,10 @@
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common;
/// <summary>Helpers for turning stored event rows into <c>HistoricalEvent</c>s.</summary>
public static class HistoricalEventRows
{
/// <summary>Clamps a stored severity into the <see cref="ushort"/> range the event carries.</summary>
/// <param name="severity">Stored severity.</param>
/// <returns>The severity clamped to 0..65535.</returns>
public static ushort ClampSeverity(int severity) => (ushort)Math.Clamp(severity, 0, ushort.MaxValue);
}
src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/HistoryUpdatePlanner.cs
+175
View File
@@ -0,0 +1,175 @@
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common;
/// <summary>What the backend did with one HistoryUpdate winner row.</summary>
public enum UpsertOutcome
{
/// <summary>The row was absent and was inserted.</summary>
Inserted,
/// <summary>The row existed and was replaced.</summary>
Replaced,
}
/// <summary>The database-independent half of one <c>UpdateDataAsync</c> call (see <see cref="HistoryUpdatePlanner.Plan"/>).</summary>
public sealed class UpdatePlan
{
internal UpdatePlan(uint?[] results, IReadOnlyList<(int Index, HistorianSampleRow Row)> winners,
long?[] keys, IReadOnlyDictionary<long, int> winnerByTick)
{
Results = results;
Winners = winners;
Keys = keys;
WinnerByTick = winnerByTick;
}
/// <summary>Pre-decided codes (invalid ts, NaN, superseded duplicate…); null = decided by the backend.</summary>
public uint?[] Results { get; }
/// <summary>Rows to send, de-duplicated per mode, in input order.</summary>
public IReadOnlyList<(int Index, HistorianSampleRow Row)> Winners { get; }
/// <summary>Per input item: its (truncated) timestamp ticks, or null when the timestamp is invalid.</summary>
internal long?[] Keys { get; }
/// <summary>(Truncated) tick → input index of the item actually sent.</summary>
internal IReadOnlyDictionary<long, int> WinnerByTick { get; }
}
/// <summary>
/// Per-item result rules for OPC UA HistoryUpdate (spec §4 "Updates"): exactly one status per input item, in input
/// order.
/// <para><b>Duplicate timestamps within one request</b> (compared at the backend's resolution): Insert — the first
/// occurrence is applied, later ones are <c>BadEntryExists</c>; Replace/Update — last wins, and every earlier
/// occurrence of that timestamp reports the same code as the winning one. DeleteAtTime — the first occurrence
/// reports the real outcome, later ones <c>BadNoEntryExists</c>.</para>
/// <para><b>Invalid items.</b> A null or <see cref="DateTime.MinValue"/> timestamp is <c>BadInvalidTimestamp</c>.
/// A Double NaN/±Infinity is rejected with <c>BadInvalidArgument</c>. An Empty (null) value is only accepted with a
/// Bad status (an operator-authored outage marker); with a Good or Uncertain status it is <c>BadInvalidArgument</c>.
/// Other items still proceed.</para>
/// <para><b>Tag names over <see cref="HistorianTagNames.MaxLength"/> characters</b> are refused, never truncated:
/// every item reports <c>BadInvalidArgument</c> and nothing is sent.</para>
/// </summary>
public static class HistoryUpdatePlanner
{
/// <summary>The pre-pass: classifies every item and picks the winner row per timestamp.</summary>
/// <param name="tag">Historian tag name.</param>
/// <param name="mode">Insert, Replace or Update.</param>
/// <param name="values">Input items, in request order.</param>
/// <param name="resolutionTicks">1 = 100 ns (SQL Server); 10 = timestamps truncated to µs BEFORE de-duplication (QuestDB).</param>
/// <returns>The plan.</returns>
public static UpdatePlan Plan(string tag, HistoryUpdateMode mode, IReadOnlyList<HistorizationValue> values,
long resolutionTicks = 1)
{
ArgumentNullException.ThrowIfNull(tag);
ArgumentNullException.ThrowIfNull(values);
ArgumentOutOfRangeException.ThrowIfLessThan(resolutionTicks, 1);
var results = new uint?[values.Count];
var winner = new Dictionary<long, int>(); // tick -> index of the item actually sent
if (!HistorianTagNames.IsValid(tag))
{
Array.Fill(results, HistorianStatusCodes.BadInvalidArgument);
return new UpdatePlan(results, [], new long?[values.Count], winner);
}
var times = new DateTime[values.Count];
var keys = new long?[values.Count];
for (var i = 0; i < values.Count; i++)
{
var v = values[i];
if (v.TimestampUtc is not { } ts || ts == DateTime.MinValue)
{
results[i] = HistorianStatusCodes.BadInvalidTimestamp;
continue;
}
times[i] = LiveWritePreparer.Truncate(ts, resolutionTicks);
keys[i] = times[i].Ticks;
if ((v.Value.Kind == HistorianValueKind.Double && !double.IsFinite(v.Value.Number))
|| (v.Value.Kind == HistorianValueKind.Empty && !StatusBits.IsBad(v.StatusCode)))
{
results[i] = HistorianStatusCodes.BadInvalidArgument;
continue;
}
var key = times[i].Ticks;
if (!winner.TryGetValue(key, out _))
winner[key] = i;
else if (mode == HistoryUpdateMode.Insert)
results[i] = HistorianStatusCodes.BadEntryExists; // first occurrence stays the winner
else
winner[key] = i; // last wins
}
var send = winner.Values.OrderBy(i => i)
.Select(i => (i, new HistorianSampleRow(times[i], values[i].Value, values[i].StatusCode)))
.ToList();
return new UpdatePlan(results, send, keys, winner);
}
/// <summary>The post-pass: fills in each winner's code from what the backend wrote, then the superseded duplicates.</summary>
/// <param name="plan">The plan from <see cref="Plan"/>.</param>
/// <param name="mode">The same mode passed to <see cref="Plan"/>.</param>
/// <param name="outcomeByTick">Ticks of each winner that the backend actually wrote → Inserted/Replaced; absent ⇒ not
/// written (Insert: the row already existed ⇒ <c>BadEntryExists</c>; Replace: no row ⇒ <c>BadNoEntryExists</c>).</param>
/// <returns>One status per input item, in input order.</returns>
public static IReadOnlyList<uint> Complete(UpdatePlan plan, HistoryUpdateMode mode,
IReadOnlyDictionary<long, UpsertOutcome> outcomeByTick)
{
ArgumentNullException.ThrowIfNull(plan);
ArgumentNullException.ThrowIfNull(outcomeByTick);
var results = (uint?[])plan.Results.Clone();
// No written row means the absent branch: Insert => matched (exists); Replace => not matched.
var absent = mode == HistoryUpdateMode.Replace
? HistorianStatusCodes.BadNoEntryExists
: HistorianStatusCodes.BadEntryExists;
foreach (var (index, row) in plan.Winners)
results[index] = outcomeByTick.TryGetValue(row.TimeUtc.Ticks, out var o)
? o == UpsertOutcome.Inserted ? HistorianStatusCodes.GoodEntryInserted : HistorianStatusCodes.GoodEntryReplaced
: absent;
var final = new uint[results.Length];
for (var i = 0; i < results.Length; i++) final[i] = results[i] ?? 0u;
// Superseded earlier duplicates (Replace/Update) share the winner's code.
if (mode != HistoryUpdateMode.Insert)
for (var i = 0; i < results.Length; i++)
if (plan.Keys[i] is { } key && results[i] is null
&& plan.WinnerByTick.TryGetValue(key, out var w) && w != i)
final[i] = final[w];
return final;
}
/// <summary>
/// DeleteAtTime results: <see cref="DateTime.MinValue"/> ⇒ <c>BadInvalidTimestamp</c>; the first occurrence of a
/// deleted time ⇒ <c>Good</c>; everything else (unknown tag, no row, later duplicate) ⇒ <c>BadNoEntryExists</c>.
/// </summary>
/// <param name="times">Requested times, in request order (already at the backend's resolution).</param>
/// <param name="deletedTicks">Ticks of the rows the backend actually deleted.</param>
/// <param name="tagKnown">Whether the tag exists.</param>
/// <returns>One status per requested time.</returns>
public static IReadOnlyList<uint> DeleteAtTimeResults(IReadOnlyList<DateTime> times, ISet<long> deletedTicks, bool tagKnown)
{
ArgumentNullException.ThrowIfNull(times);
ArgumentNullException.ThrowIfNull(deletedTicks);
var results = new uint[times.Count];
var first = new Dictionary<long, int>();
for (var i = 0; i < times.Count; i++)
{
results[i] = HistorianStatusCodes.BadNoEntryExists;
if (times[i] == DateTime.MinValue) results[i] = HistorianStatusCodes.BadInvalidTimestamp;
else first.TryAdd(times[i].Ticks, i);
}
if (!tagKnown) return results;
foreach (var (ticks, idx) in first)
if (deletedTicks.Contains(ticks))
results[idx] = HistorianStatusCodes.Good;
return results;
}
/// <summary>DeleteRaw result: unknown tag or nothing deleted ⇒ <c>BadNoData</c>; otherwise <c>Good</c>.</summary>
/// <param name="tagKnown">Whether the tag exists.</param>
/// <param name="rowsAffected">Rows the backend deleted.</param>
/// <returns>The status.</returns>
public static uint DeleteRawResult(bool tagKnown, long rowsAffected) =>
tagKnown && rowsAffected > 0 ? HistorianStatusCodes.Good : HistorianStatusCodes.BadNoData;
}
src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/LiveWritePreparer.cs
+54
View File
@@ -0,0 +1,54 @@
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common;
/// <summary>
/// Normalises a live-value batch for storage. <b>Non-finite doubles.</b> NaN / ±Infinity are stored as
/// <see cref="HistorianValueKind.Empty"/> so the sample's presence survives. <b>An Empty sample is always Bad</b>:
/// a non-finite Double whose status is not already Bad is stored with <c>BadOutOfRange</c>; any other Empty with a
/// non-Bad status is stored with <c>BadNoData</c>; an existing Bad code is kept. <b>Duplicate timestamps</b> within
/// a batch collapse to the first (matching insert-if-absent across batches).
/// </summary>
public static class LiveWritePreparer
{
/// <summary>
/// Normalises a batch: null timestamp ⇒ now, timestamps truncated to <paramref name="resolutionTicks"/>,
/// non-finite doubles ⇒ Empty, every Empty forced Bad, and duplicate (truncated) timestamps collapsed to the first.
/// </summary>
/// <param name="values">Input samples, in append order.</param>
/// <param name="utcNow">Clock for null timestamps.</param>
/// <param name="resolutionTicks">1 = keep 100 ns (SQL Server); 10 = truncate to µs BEFORE de-duplication (QuestDB).</param>
/// <returns>Time-unique rows.</returns>
public static List<HistorianSampleRow> Prepare(IReadOnlyList<HistorizationValue> values, Func<DateTime> utcNow,
long resolutionTicks = 1)
{
ArgumentOutOfRangeException.ThrowIfLessThan(resolutionTicks, 1);
var seen = new HashSet<long>();
var rows = new List<HistorianSampleRow>(values.Count);
foreach (var v in values)
{
var t = v.TimestampUtc ?? utcNow();
t = t.Kind == DateTimeKind.Local ? t.ToUniversalTime() : DateTime.SpecifyKind(t, DateTimeKind.Utc);
t = Truncate(t, resolutionTicks);
if (!seen.Add(t.Ticks)) continue;
var value = v.Value;
var status = v.StatusCode;
if (value.Kind == HistorianValueKind.Double && !double.IsFinite(value.Number))
{
value = HistorianValue.Empty;
if (!StatusBits.IsBad(status)) status = HistorianStatusCodes.BadOutOfRange;
}
else if (value.Kind == HistorianValueKind.Empty && !StatusBits.IsBad(status))
{
status = HistorianStatusCodes.BadNoData;
}
rows.Add(new HistorianSampleRow(t, value, status));
}
return rows;
}
/// <summary>Truncates <paramref name="t"/> down to a multiple of <paramref name="resolutionTicks"/>, keeping its kind.</summary>
internal static DateTime Truncate(DateTime t, long resolutionTicks) =>
resolutionTicks == 1 ? t : t.AddTicks(-(t.Ticks % resolutionTicks));
}
src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/SampleColumns.cs
+66
View File
@@ -0,0 +1,66 @@
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common;
/// <summary>The (ValueKind, NumValue, IntValue, StrValue) column quad for a value.</summary>
/// <param name="ValueKind">The <see cref="HistorianValueKind"/> as a byte.</param>
/// <param name="NumValue">Numeric column (null = NULL).</param>
/// <param name="IntValue">Integer column (null = NULL).</param>
/// <param name="StrValue">String column (null = NULL).</param>
public readonly record struct SampleColumns(byte ValueKind, double? NumValue, long? IntValue, string? StrValue);
/// <summary>
/// HistorianValue ⇄ column mapping (spec §4). Mapping: Double → Num; Int64 → Int + Num(double);
/// Boolean → Int 0/1 + Num 0/1; String → Str; DateTime → Int (ticks); Empty → all NULL.
/// </summary>
public static class SampleColumnMapper
{
/// <summary>Splits a value into its column quad.</summary>
/// <param name="v">Typed value.</param>
/// <returns>The columns to store.</returns>
public static SampleColumns ToColumns(HistorianValue v) => v.Kind switch
{
HistorianValueKind.Double => new((byte)v.Kind, v.Number, null, null),
HistorianValueKind.Int64 => new((byte)v.Kind, v.Number, v.Integer, null),
HistorianValueKind.Boolean => new((byte)v.Kind, v.Integer != 0 ? 1d : 0d, v.Integer != 0 ? 1L : 0L, null),
HistorianValueKind.String => new((byte)v.Kind, null, null, v.Text ?? ""),
HistorianValueKind.DateTime => new((byte)v.Kind, null, v.Integer, null),
_ => new((byte)HistorianValueKind.Empty, null, null, null),
};
/// <summary>Rebuilds a value from stored columns.</summary>
/// <param name="valueKind">ValueKind column.</param>
/// <param name="num">NumValue column (null = NULL).</param>
/// <param name="integer">IntValue column (null = NULL).</param>
/// <param name="str">StrValue column (null = NULL).</param>
/// <returns>The typed value; an unknown kind yields <see cref="HistorianValue.Empty"/>.</returns>
public static HistorianValue FromColumns(byte valueKind, double? num, long? integer, string? str) => (HistorianValueKind)valueKind switch
{
HistorianValueKind.Double => HistorianValue.FromDouble(num ?? 0d),
HistorianValueKind.Int64 => HistorianValue.FromInt64(integer ?? 0L),
HistorianValueKind.Boolean => HistorianValue.FromBoolean((integer ?? 0L) != 0),
HistorianValueKind.String => HistorianValue.FromString(str ?? ""),
HistorianValueKind.DateTime => new HistorianValue(HistorianValueKind.DateTime, 0d, integer ?? 0L, null),
_ => HistorianValue.Empty,
};
/// <summary>Stores a uint status bit-for-bit in an int column.</summary>
/// <param name="status">Status code.</param>
/// <returns>Same bits as int.</returns>
public static int ToDbStatus(uint status) => unchecked((int)status);
/// <summary>Reads an int status column back to a uint code.</summary>
/// <param name="dbStatus">Column value.</param>
/// <returns>Same bits as uint.</returns>
public static uint FromDbStatus(int dbStatus) => unchecked((uint)dbStatus);
/// <summary>Stores a uint status in a 64-bit column (zero-extended, so it is never negative).</summary>
/// <param name="status">Status code.</param>
/// <returns>The status as a non-negative long.</returns>
public static long ToLongStatus(uint status) => status;
/// <summary>Reads a 64-bit status column back to a uint code (low 32 bits).</summary>
/// <param name="dbStatus">Column value.</param>
/// <returns>The status code.</returns>
public static uint FromLongStatus(long dbStatus) => unchecked((uint)dbStatus);
}
src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/StatusBits.cs
+33
View File
@@ -0,0 +1,33 @@
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common;
/// <summary>
/// Bit masks over a 32-bit OPC UA status code (Part 4 §7.39). These are masks, not emitted codes, so their
/// names deliberately avoid the Good/Bad/Uncertain prefix the status-code parity scan keys on.
/// </summary>
public static class StatusBits
{
/// <summary>Severity + sub-code (top 16 bits) of a status, i.e. the code without info bits.</summary>
public const uint CodeBitsMask = 0xFFFF0000u;
/// <summary>The two severity bits (Good = 00, Uncertain = 01, Bad = 10).</summary>
public const uint SeverityMask = 0xC0000000u;
/// <summary>The severity bit set on every Bad code.</summary>
public const uint SeverityBadBit = 0x80000000u;
/// <summary>The severity bit set on every Uncertain code (bit 30).</summary>
public const uint SeverityUncertainBit = 0x40000000u;
/// <summary>The two historian info bits (Calculated, Interpolated) in the low bits of the info field.</summary>
public const uint HistorianBitsMask = 0x3u;
/// <summary>True when <paramref name="status"/> has Bad severity.</summary>
/// <param name="status">A status code.</param>
/// <returns>Whether the Bad severity bit is set.</returns>
public static bool IsBad(uint status) => (status & SeverityBadBit) != 0;
/// <summary>True when <paramref name="status"/> has Good severity.</summary>
/// <param name="status">A status code.</param>
/// <returns>Whether both severity bits are clear.</returns>
public static bool IsGood(uint status) => (status & SeverityMask) == 0;
}
src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/StoredSample.cs
+32
View File
@@ -0,0 +1,32 @@
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common;
/// <summary>A stored sample as read back: source time, typed value and stored status.</summary>
/// <param name="TimeUtc">Source timestamp (UTC).</param>
/// <param name="Value">Typed value (<see cref="HistorianValue.Empty"/> for an outage marker).</param>
/// <param name="Status">Stored OPC UA status code.</param>
public readonly record struct StoredSample(DateTime TimeUtc, HistorianValue Value, uint Status)
{
/// <summary>Severity Good or Uncertain, and a value present (an Empty sample is an outage marker).</summary>
public bool IsUsable => !StatusBits.IsBad(Status) && Value.Kind != HistorianValueKind.Empty;
/// <summary>Usable with severity Good.</summary>
public bool IsGood => IsUsable && StatusBits.IsGood(Status);
/// <summary>
/// Builds a sample from stored columns, applying the quality clamp: an Empty (no-value) row whose stored status
/// is not Bad is served as <c>BadNoData</c> — defence in depth for rows written before the writer forced every
/// Empty sample Bad.
/// </summary>
/// <param name="timeUtc">Source timestamp (UTC).</param>
/// <param name="value">Stored value.</param>
/// <param name="status">Stored status code.</param>
/// <returns>The sample, clamped.</returns>
public static StoredSample FromStored(DateTime timeUtc, HistorianValue value, uint status)
{
if (value.Kind == HistorianValueKind.Empty && !StatusBits.IsBad(status))
status = HistorianStatusCodes.BadNoData;
return new StoredSample(timeUtc, value, status);
}
}
src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.csproj
+22
View File
@@ -0,0 +1,22 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
<GenerateDocumentationFile>true</GenerateDocumentationFile>
<NoWarn>$(NoWarn);CS1591</NoWarn>
<Platforms>AnyCPU;x64</Platforms>
<RootNamespace>ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common</RootNamespace>
</PropertyGroup>
<!--
The database-independent core shared by the SQL historian backends (SQL Server, QuestDB): status codes and bits,
value/column mapping, the AtTime and aggregate evaluators, bucket planning, live-write preparation, HistoryUpdate
result planning and the bootstrap failure cache. NO database client and no I/O.
-->
<ItemGroup>
<ProjectReference Include="..\..\Core\ZB.MOM.WW.OtOpcUa.Core.Abstractions\ZB.MOM.WW.OtOpcUa.Core.Abstractions.csproj"/>
<!-- AlarmEventIds.ComputeEventId takes an AlarmHistorianEvent (same signature as the SQL Server writer had). -->
<ProjectReference Include="..\..\Core\ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian\ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian.csproj"/>
</ItemGroup>
</Project>
src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SampleRowMapper.cs
+6 -55
View File
@@ -1,22 +1,12 @@
using System.Data;
using Microsoft.Data.SqlClient.Server;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer;
/// <summary>One sample bound for a <c>SampleRow</c> TVP.</summary>
/// <param name="TimeUtc">Source timestamp (UTC, full tick precision).</param>
/// <param name="Value">Typed value.</param>
/// <param name="StatusCode">OPC UA status code (stored bit-for-bit as int).</param>
internal readonly record struct SqlSampleRow(DateTime TimeUtc, HistorianValue Value, uint StatusCode);
/// <summary>The (ValueKind, NumValue, IntValue, StrValue) column quad for a value.</summary>
internal readonly record struct SampleColumns(byte ValueKind, double? NumValue, long? IntValue, string? StrValue);
/// <summary>
/// HistorianValue ⇄ column mapping (spec §4) and the <c>SqlDataRecord</c> TVP builders. Mapping:
/// Double → Num; Int64 → Int + Num(double); Boolean → Int 0/1 + Num 0/1; String → Str; DateTime → Int (ticks);
/// Empty → all NULL.
/// The <c>SqlDataRecord</c> TVP builders. The HistorianValue ⇄ column mapping itself is
/// <see cref="SampleColumnMapper"/> (shared with the other SQL backends).
/// </summary>
internal static class SampleRowMapper
{
@@ -32,60 +22,21 @@ internal static class SampleRowMapper
private static readonly SqlMetaData[] TimeMeta = [new("T", SqlDbType.DateTime2, 0, 7)];
/// <summary>Splits a value into its column quad.</summary>
/// <param name="v">Typed value.</param>
/// <returns>The columns to store.</returns>
public static SampleColumns ToColumns(HistorianValue v) => v.Kind switch
{
HistorianValueKind.Double => new((byte)v.Kind, v.Number, null, null),
HistorianValueKind.Int64 => new((byte)v.Kind, v.Number, v.Integer, null),
HistorianValueKind.Boolean => new((byte)v.Kind, v.Integer != 0 ? 1d : 0d, v.Integer != 0 ? 1L : 0L, null),
HistorianValueKind.String => new((byte)v.Kind, null, null, v.Text ?? ""),
HistorianValueKind.DateTime => new((byte)v.Kind, null, v.Integer, null),
_ => new((byte)HistorianValueKind.Empty, null, null, null),
};
/// <summary>Rebuilds a value from stored columns.</summary>
/// <param name="valueKind">ValueKind column.</param>
/// <param name="num">NumValue column (null = NULL).</param>
/// <param name="integer">IntValue column (null = NULL).</param>
/// <param name="str">StrValue column (null = NULL).</param>
/// <returns>The typed value; an unknown kind yields <see cref="HistorianValue.Empty"/>.</returns>
public static HistorianValue FromColumns(byte valueKind, double? num, long? integer, string? str) => (HistorianValueKind)valueKind switch
{
HistorianValueKind.Double => HistorianValue.FromDouble(num ?? 0d),
HistorianValueKind.Int64 => HistorianValue.FromInt64(integer ?? 0L),
HistorianValueKind.Boolean => HistorianValue.FromBoolean((integer ?? 0L) != 0),
HistorianValueKind.String => HistorianValue.FromString(str ?? ""),
HistorianValueKind.DateTime => new HistorianValue(HistorianValueKind.DateTime, 0d, integer ?? 0L, null),
_ => HistorianValue.Empty,
};
/// <summary>Stores a uint status bit-for-bit in the int column.</summary>
/// <param name="status">Status code.</param>
/// <returns>Same bits as int.</returns>
public static int ToDbStatus(uint status) => unchecked((int)status);
/// <summary>Reads the int status column back to a uint code.</summary>
/// <param name="dbStatus">Column value.</param>
/// <returns>Same bits as uint.</returns>
public static uint FromDbStatus(int dbStatus) => unchecked((uint)dbStatus);
/// <summary>Builds <c>SampleRow</c> TVP records (datetime2(7); never a DataTable). Lazy; enumerate once.</summary>
/// <param name="rows">Samples; must be time-unique (the TVP has a PK on time) and non-empty when bound.</param>
/// <returns>Records for <see cref="SqlHistorianConnection.AddStructured"/>.</returns>
public static IEnumerable<SqlDataRecord> ToSampleRecords(IEnumerable<SqlSampleRow> rows)
public static IEnumerable<SqlDataRecord> ToSampleRecords(IEnumerable<HistorianSampleRow> rows)
{
foreach (var r in rows)
{
var c = ToColumns(r.Value);
var c = SampleColumnMapper.ToColumns(r.Value);
var rec = new SqlDataRecord(SampleMeta);
rec.SetDateTime(0, DateTime.SpecifyKind(r.TimeUtc, DateTimeKind.Utc));
rec.SetByte(1, c.ValueKind);
if (c.NumValue is { } n) rec.SetDouble(2, n); else rec.SetDBNull(2);
if (c.IntValue is { } i) rec.SetInt64(3, i); else rec.SetDBNull(3);
if (c.StrValue is { } s) rec.SetString(4, s); else rec.SetDBNull(4);
rec.SetInt32(5, ToDbStatus(r.StatusCode));
rec.SetInt32(5, SampleColumnMapper.ToDbStatus(r.StatusCode));
yield return rec;
}
}
src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlAggregateQueries.cs
+3 -28
View File
@@ -1,22 +1,10 @@
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer;
/// <summary>The SQL shape that serves an aggregate. One query per family.</summary>
internal enum SqlAggregateFamily
{
/// <summary>Average / Minimum / Maximum / Total / Count / Range / PercentGood / PercentBad: one GROUP BY.</summary>
Simple,
/// <summary>Start / End / Delta: first + last usable sample per bucket (<c>ROW_NUMBER</c>).</summary>
StartEnd,
/// <summary>MinimumActualTime / MaximumActualTime: extreme usable sample per bucket, ties ⇒ earliest.</summary>
ActualTime,
}
/// <summary>
/// Push-down T-SQL for processed (aggregate) reads (spec §4 "Processed"). Every query takes
/// Push-down T-SQL for processed (aggregate) reads (spec §4 "Processed"), one query per
/// <see cref="AggregateFamily"/> (<see cref="AggregateFamilies.FamilyOf"/> picks it). Every query takes
/// <c>@id</c> (int TagId), <c>@start</c>/<c>@end</c> (<c>datetime2(7)</c>, the forward window
/// <c>[start, end)</c>) and <c>@intervalTicks</c> (bigint bucket width in 100 ns ticks), and returns one row
/// per <b>non-empty</b> bucket. The caller generates every interval and fills the empty ones.
@@ -52,19 +40,6 @@ internal static class SqlAggregateQueries
/// <summary><c>ValueKind</c> of String (4) and DateTime (5): usable but not numeric.</summary>
private const string NonNumericKinds = "4, 5";
/// <summary>The family that serves <paramref name="aggregate"/>, or null when this backend does not.</summary>
/// <param name="aggregate">Requested aggregate.</param>
/// <returns>The family, or null for an unsupported aggregate.</returns>
public static SqlAggregateFamily? FamilyOf(HistoryAggregateType aggregate) => aggregate switch
{
HistoryAggregateType.Average or HistoryAggregateType.Minimum or HistoryAggregateType.Maximum
or HistoryAggregateType.Total or HistoryAggregateType.Count or HistoryAggregateType.Range
or HistoryAggregateType.PercentGood or HistoryAggregateType.PercentBad => SqlAggregateFamily.Simple,
HistoryAggregateType.Start or HistoryAggregateType.End or HistoryAggregateType.Delta => SqlAggregateFamily.StartEnd,
HistoryAggregateType.MinimumActualTime or HistoryAggregateType.MaximumActualTime => SqlAggregateFamily.ActualTime,
_ => null,
};
/// <summary>
/// Average / Minimum / Maximum / Total / Count / Range / PercentGood / PercentBad.
/// Family columns: <c>6 AvgValue, 7 MinValue, 8 MaxValue, 9 SumValue</c> (float, NULL when no usable
src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlHistorianStatusCodes.cs
-81
View File
@@ -1,81 +0,0 @@
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer;
/// <summary>
/// Every OPC UA status this backend emits, as named constants (inline literals are invisible to
/// <c>StatusCodeParityTests</c>). Values mirror <c>Opc.Ua.StatusCodes</c>.
/// </summary>
internal static class SqlHistorianStatusCodes
{
public const uint Good = 0x00000000u;
public const uint GoodEntryInserted = 0x00A20000u;
public const uint GoodEntryReplaced = 0x00A30000u;
public const uint BadNoData = 0x809B0000u;
public const uint BadEntryExists = 0x809F0000u;
public const uint BadNoEntryExists = 0x80A00000u;
public const uint BadInvalidTimestamp = 0x80230000u;
public const uint BadAggregateInvalidInputs = 0x80D60000u;
public const uint UncertainDataSubNormal = 0x40A40000u;
/// <summary>Stored for a non-finite Double (NaN / ±Infinity) that arrived with a non-Bad status.</summary>
public const uint BadOutOfRange = 0x803C0000u;
}
/// <summary>
/// Bit masks over a 32-bit OPC UA status code (Part 4 §7.39). These are masks, not emitted codes, so their
/// names deliberately avoid the Good/Bad/Uncertain prefix the status-code parity scan keys on.
/// </summary>
internal static class StatusBits
{
/// <summary>Severity + sub-code (top 16 bits) of a status, i.e. the code without info bits.</summary>
public const uint CodeBitsMask = 0xFFFF0000u;
/// <summary>The two severity bits (Good = 00, Uncertain = 01, Bad = 10).</summary>
public const uint SeverityMask = 0xC0000000u;
/// <summary>The severity bit set on every Bad code.</summary>
public const uint SeverityBadBit = 0x80000000u;
/// <summary>The two historian info bits (Calculated, Interpolated) in the low bits of the info field.</summary>
public const uint HistorianBitsMask = 0x3u;
/// <summary>True when <paramref name="status"/> has Bad severity.</summary>
/// <param name="status">A status code.</param>
/// <returns>Whether the Bad severity bit is set.</returns>
public static bool IsBad(uint status) => (status & SeverityBadBit) != 0;
/// <summary>True when <paramref name="status"/> has Good severity.</summary>
/// <param name="status">A status code.</param>
/// <returns>Whether both severity bits are clear.</returns>
public static bool IsGood(uint status) => (status & SeverityMask) == 0;
}
/// <summary>Part 4 §7.39 historian info bits OR-ed onto a status code.</summary>
internal static class HistorianInfoBits
{
/// <summary>InfoType = DataValue.</summary>
public const uint InfoTypeDataValue = 0x00000400u;
/// <summary>The value was produced by an aggregate calculation.</summary>
public const uint HistorianCalculated = 0x1u;
/// <summary>The value was interpolated.</summary>
public const uint HistorianInterpolated = 0x2u;
/// <summary>
/// Marks <paramref name="status"/> as a Calculated data value. Any historian bits the status already carries
/// (e.g. a stored Interpolated sample) are cleared first, so the result is never both.
/// </summary>
/// <param name="status">Base status code.</param>
/// <returns>Status with InfoType and Calculated bits set.</returns>
public static uint WithCalculated(uint status) =>
(status & ~StatusBits.HistorianBitsMask) | InfoTypeDataValue | HistorianCalculated;
/// <summary>
/// Marks <paramref name="status"/> as an Interpolated data value. Any historian bits the status already
/// carries (e.g. a stored Calculated sample) are cleared first, so the result is never both.
/// </summary>
/// <param name="status">Base status code.</param>
/// <returns>Status with InfoType and Interpolated bits set.</returns>
public static uint WithInterpolated(uint status) =>
(status & ~StatusBits.HistorianBitsMask) | InfoTypeDataValue | HistorianInterpolated;
}
src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerAlarmHistorianWriter.cs
+4 -51
View File
@@ -1,17 +1,16 @@
using System.Data;
using System.Security.Cryptography;
using System.Text;
using Microsoft.Data.SqlClient;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Logging.Abstractions;
using ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer;
/// <summary>
/// <see cref="IAlarmHistorianWriter"/> over the SQL <c>Event</c> table. Per event: insert-if-absent keyed by a
/// deterministic <see cref="ComputeEventId"/>, so the store-and-forward drain's at-least-once redelivery (and both
/// deterministic <see cref="AlarmEventIds.ComputeEventId"/>, so the store-and-forward drain's at-least-once redelivery (and both
/// pair nodes writing the same event) converge on one row. Outcomes: Ack on insert or already-present;
/// RetryPlease on transient SQL / schema-bootstrap / unclassified SQL errors and on cancellation; PermanentFail only
/// for data errors (truncation, conversion, null, range, constraint); unknown/non-SQL exceptions retry. Never throws.
@@ -76,12 +75,12 @@ internal sealed class SqlServerAlarmHistorianWriter : IAlarmHistorianWriter
{
await using var conn = await _conn.OpenAsync(ct).ConfigureAwait(false);
await using var cmd = _conn.CreateCommand(conn, sql);
cmd.Parameters.Add(new SqlParameter("@id", SqlDbType.NVarChar, 64) { Value = ComputeEventId(e) });
cmd.Parameters.Add(new SqlParameter("@id", SqlDbType.NVarChar, 64) { Value = AlarmEventIds.ComputeEventId(e) });
cmd.Parameters.Add(new SqlParameter("@src", SqlDbType.NVarChar, -1) { Value = e.EquipmentPath ?? "" });
cmd.Parameters.Add(new SqlParameter("@aid", SqlDbType.NVarChar, -1) { Value = e.AlarmId ?? "" });
cmd.Parameters.Add(new SqlParameter("@an", SqlDbType.NVarChar, -1) { Value = e.AlarmName ?? "" });
cmd.Parameters.Add(new SqlParameter("@at", SqlDbType.NVarChar, -1) { Value = e.AlarmTypeName ?? "" });
cmd.Parameters.Add(new SqlParameter("@sev", SqlDbType.Int) { Value = ToNumericSeverity(e.Severity) });
cmd.Parameters.Add(new SqlParameter("@sev", SqlDbType.Int) { Value = AlarmEventIds.ToNumericSeverity(e.Severity) });
cmd.Parameters.Add(new SqlParameter("@ek", SqlDbType.NVarChar, -1) { Value = e.EventKind ?? "" });
cmd.Parameters.Add(new SqlParameter("@msg", SqlDbType.NVarChar, -1) { Value = (object?)e.Message ?? DBNull.Value });
cmd.Parameters.Add(new SqlParameter("@usr", SqlDbType.NVarChar, -1) { Value = (object?)e.User ?? DBNull.Value });
@@ -116,50 +115,4 @@ internal sealed class SqlServerAlarmHistorianWriter : IAlarmHistorianWriter
return HistorianWriteOutcome.PermanentFail;
return HistorianWriteOutcome.RetryPlease;
}
// Same numeric Part 9 priorities the gateway path sends (AlarmEventMapper.ToNumericSeverity).
private static int ToNumericSeverity(AlarmSeverity s) => s switch
{
AlarmSeverity.Low => 200,
AlarmSeverity.Medium => 500,
AlarmSeverity.High => 700,
AlarmSeverity.Critical => 900,
_ => 500,
};
/// <summary>
/// Lowercase-hex SHA-256 over length-prefixed UTF-8 fields (little-endian int32 byte length, -1 for null):
/// EquipmentPath, AlarmId, EventKind, TimestampUtc.Ticks (decimal), User, Comment, Message. Length-prefixing keeps
/// field boundaries unambiguous (<c>"a|b"+"c"</c> ≠ <c>"a"+"b|c"</c>).
/// </summary>
/// <param name="e">The event.</param>
/// <returns>64 hex characters.</returns>
internal static string ComputeEventId(AlarmHistorianEvent e)
{
using var ms = new MemoryStream();
Field(ms, e.EquipmentPath);
Field(ms, e.AlarmId);
Field(ms, e.EventKind);
Field(ms, e.TimestampUtc.Ticks.ToString(System.Globalization.CultureInfo.InvariantCulture));
Field(ms, e.User);
Field(ms, e.Comment);
Field(ms, e.Message);
return Convert.ToHexStringLower(SHA256.HashData(ms.ToArray()));
}
private static void Field(MemoryStream ms, string? value)
{
Span<byte> len = stackalloc byte[4];
if (value is null)
{
System.Buffers.Binary.BinaryPrimitives.WriteInt32LittleEndian(len, -1);
ms.Write(len);
return;
}
var bytes = Encoding.UTF8.GetBytes(value);
System.Buffers.Binary.BinaryPrimitives.WriteInt32LittleEndian(len, bytes.Length);
ms.Write(len);
ms.Write(bytes);
}
}
src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerHistorianDataSource.cs
+34 -196
View File
@@ -4,6 +4,7 @@ using Microsoft.Data.SqlClient;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Logging.Abstractions;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer;
@@ -39,7 +40,7 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer;
/// on Raw and AtTime alike.
/// </para>
/// <para>
/// <b>Long names.</b> A tag name or event source name longer than <see cref="SqlServerTagCatalog.MaxNameLength"/>
/// <b>Long names.</b> A tag name or event source name longer than <see cref="HistorianTagNames.MaxLength"/>
/// can never be stored, so it is answered as not found (Raw: empty; AtTime / Processed: <c>BadNoData</c>;
/// Events: none) without being truncated onto another series.
/// </para>
@@ -47,7 +48,7 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer;
public sealed class SqlServerHistorianDataSource : IHistorianDataSource
{
/// <summary>Spec §9.7: a processed read covering more intervals than this is refused.</summary>
public const int MaxProcessedBuckets = 100_000;
public const int MaxProcessedBuckets = BucketPlan.MaxBuckets;
private const string SampleColumns = "SourceTimeUtc, ValueKind, NumValue, IntValue, StrValue, StatusCode, ServerTimeUtc";
@@ -57,14 +58,7 @@ public sealed class SqlServerHistorianDataSource : IHistorianDataSource
private readonly ILogger _log;
private readonly TimeProvider _time;
private readonly object _healthLock = new();
private DateTime? _lastSuccessUtc;
private DateTime? _lastFailureUtc;
private string? _lastError;
private long _totalQueries;
private long _totalSuccesses;
private long _totalFailures;
private int _consecutiveFailures;
private readonly HistorianHealthCounters _health = new();
/// <summary>Creates the data source over the provider's shared connection, bootstrapper and tag catalog.</summary>
/// <param name="conn">Connection helper.</param>
@@ -192,7 +186,7 @@ public sealed class SqlServerHistorianDataSource : IHistorianDataSource
StoredSample? p = reader.IsDBNull(1) ? null : ReadStored(reader, 1);
StoredSample? n = reader.IsDBNull(8) ? null : ReadStored(reader, 8);
var serverTime = p is { } ps && ps.TimeUtc == t ? SqlHistorianConnection.GetUtc(reader, 7) : now;
byTicks[t.Ticks] = AtTimeValue(t, p, n, serverTime, now);
byTicks[t.Ticks] = AtTimeEvaluator.Evaluate(t, p, n, serverTime, now);
}
}
@@ -202,7 +196,7 @@ public sealed class SqlServerHistorianDataSource : IHistorianDataSource
var t = DateTime.SpecifyKind(timestampsUtc[i], DateTimeKind.Utc);
result[i] = byTicks.TryGetValue(t.Ticks, out var v)
? v
: new DataValueSnapshot(null, SqlHistorianStatusCodes.BadNoData, t, now);
: new DataValueSnapshot(null, HistorianStatusCodes.BadNoData, t, now);
}
RecordSuccess();
@@ -215,44 +209,6 @@ public sealed class SqlServerHistorianDataSource : IHistorianDataSource
}
}
/// <summary>Applies the AtTime rules (see <see cref="ReadAtTimeAsync"/>) to one requested time.</summary>
private static DataValueSnapshot AtTimeValue(DateTime t, StoredSample? prev, StoredSample? next, DateTime serverTime, DateTime now)
{
if (prev is not { } p)
return new DataValueSnapshot(null, SqlHistorianStatusCodes.BadNoData, t, serverTime);
if (p.TimeUtc == t)
return new DataValueSnapshot(p.Value.ToObject(), p.Status, t, serverTime);
// Inside an outage: report the outage's own Bad code, never an upgraded Uncertain.
if (!p.IsUsable)
return new DataValueSnapshot(null, p.Status & StatusBits.CodeBitsMask, t, serverTime);
// Usable previous, Bad next: hold the previous value up to the outage; do not interpolate into it.
if (next is { IsUsable: false })
return new DataValueSnapshot(p.Value.ToObject(),
HistorianInfoBits.WithInterpolated(SqlHistorianStatusCodes.UncertainDataSubNormal), t, serverTime);
if (next is { } n && p.Value.Kind is HistorianValueKind.Double or HistorianValueKind.Int64
&& n.Value.TryGetDouble(out var nv))
{
var fraction = (double)(t.Ticks - p.TimeUtc.Ticks) / (n.TimeUtc.Ticks - p.TimeUtc.Ticks);
var v = p.Value.Number + (nv - p.Value.Number) * fraction;
object value = p.Value.Kind == HistorianValueKind.Int64
? (long)Math.Round(v, MidpointRounding.AwayFromZero)
: v;
var status = p.IsGood && n.IsGood ? SqlHistorianStatusCodes.Good : SqlHistorianStatusCodes.UncertainDataSubNormal;
return new DataValueSnapshot(value, HistorianInfoBits.WithInterpolated(status), t, serverTime);
}
// Beyond the last sample AND beyond now: the value is held into the future, so it is not known to be Good.
if (next is null && t > now)
return new DataValueSnapshot(p.Value.ToObject(),
HistorianInfoBits.WithInterpolated(SqlHistorianStatusCodes.UncertainDataSubNormal), t, serverTime);
// Stepped (non-numeric kind, or after the last sample): hold the previous value and its status.
return new DataValueSnapshot(p.Value.ToObject(), HistorianInfoBits.WithInterpolated(p.Status), t, serverTime);
}
// ------------------------------------------------------------------------------------------ Events
/// <inheritdoc />
@@ -269,7 +225,7 @@ public sealed class SqlServerHistorianDataSource : IHistorianDataSource
{
// No stored event can carry a source longer than the column (the alarm writer dead-letters it rather than
// truncating), so an over-long filter matches nothing — never a truncated prefix of someone else's source.
if (sourceName is not null && sourceName.Length > SqlServerTagCatalog.MaxNameLength)
if (sourceName is not null && sourceName.Length > HistorianTagNames.MaxLength)
return new HistoricalEventsResult([], null);
try
{
@@ -301,10 +257,10 @@ public sealed class SqlServerHistorianDataSource : IHistorianDataSource
SqlHistorianConnection.GetUtc(reader, 2),
SqlHistorianConnection.GetUtc(reader, 3),
reader.IsDBNull(4) ? null : reader.GetString(4),
(ushort)Math.Clamp(reader.GetInt32(5), 0, ushort.MaxValue));
HistoricalEventRows.ClampSeverity(reader.GetInt32(5)));
if (bounded && events.Count == maxEvents)
{
continuation = Encoding.UTF8.GetBytes($"{e.EventTimeUtc.Ticks}|{e.EventId}");
continuation = Encoding.UTF8.GetBytes(EventContinuation.Encode(e.EventTimeUtc, e.EventId));
break;
}
events.Add(e);
@@ -352,57 +308,38 @@ public sealed class SqlServerHistorianDataSource : IHistorianDataSource
HistoryAggregateType aggregate, CancellationToken cancellationToken)
{
ArgumentNullException.ThrowIfNull(fullReference);
var family = SqlAggregateQueries.FamilyOf(aggregate) ?? throw new HistoryAggregateNotSupportedException(aggregate);
var reverse = endUtc < startUtc;
var (start, end) = reverse ? (endUtc, startUtc) : (startUtc, endUtc);
var windowTicks = end.Ticks - start.Ticks;
// Everything stays in 100 ns ticks (≤ ≈3.2e18 across 0001–9999, inside a long) — no ns scaling that could
// wrap. An interval wider than the window is one bucket, so clamp it to the window; the ceiling division
// is written without "+ interval - 1" so a TimeSpan near MaxValue cannot overflow either.
var intervalTicks = interval > TimeSpan.Zero ? Math.Min(interval.Ticks, windowTicks) : windowTicks;
long bucketCount = windowTicks == 0 ? 0 : windowTicks / intervalTicks + (windowTicks % intervalTicks == 0 ? 0 : 1);
if (bucketCount > MaxProcessedBuckets)
throw new ArgumentOutOfRangeException(nameof(interval), bucketCount,
$"A processed read may cover at most {MaxProcessedBuckets} intervals; this window holds {bucketCount}.");
var family = AggregateFamilies.FamilyOf(aggregate) ?? throw new HistoryAggregateNotSupportedException(aggregate);
var plan = BucketPlan.Create(startUtc, endUtc, interval);
try
{
var now = _time.GetUtcNow().UtcDateTime;
var rows = new Dictionary<long, BucketRow>();
var tagId = bucketCount == 0 ? null : await ResolveAsync(fullReference, cancellationToken).ConfigureAwait(false);
var evaluated = new Dictionary<long, DataValueSnapshot>();
var tagId = plan.BucketCount == 0 ? null : await ResolveAsync(fullReference, cancellationToken).ConfigureAwait(false);
if (tagId is { } id)
{
var sample = _conn.Q("Sample");
var sql = family switch
{
SqlAggregateFamily.Simple => SqlAggregateQueries.Simple(sample),
SqlAggregateFamily.StartEnd => SqlAggregateQueries.StartEnd(sample),
AggregateFamily.Simple => SqlAggregateQueries.Simple(sample),
AggregateFamily.StartEnd => SqlAggregateQueries.StartEnd(sample),
_ => SqlAggregateQueries.ActualTime(sample, aggregate == HistoryAggregateType.MaximumActualTime),
};
await using var conn = await _conn.OpenAsync(cancellationToken).ConfigureAwait(false);
await using var cmd = _conn.CreateCommand(conn, sql);
cmd.Parameters.Add(new SqlParameter("@id", SqlDbType.Int) { Value = id });
SqlHistorianConnection.AddDateTime2(cmd, "@start", DateTime.SpecifyKind(start, DateTimeKind.Utc));
SqlHistorianConnection.AddDateTime2(cmd, "@end", DateTime.SpecifyKind(end, DateTimeKind.Utc));
cmd.Parameters.Add(new SqlParameter("@intervalTicks", SqlDbType.BigInt) { Value = intervalTicks });
SqlHistorianConnection.AddDateTime2(cmd, "@start", DateTime.SpecifyKind(plan.StartUtc, DateTimeKind.Utc));
SqlHistorianConnection.AddDateTime2(cmd, "@end", DateTime.SpecifyKind(plan.EndUtc, DateTimeKind.Utc));
cmd.Parameters.Add(new SqlParameter("@intervalTicks", SqlDbType.BigInt) { Value = plan.IntervalTicks });
await using var reader = await cmd.ExecuteReaderAsync(cancellationToken).ConfigureAwait(false);
while (await reader.ReadAsync(cancellationToken).ConfigureAwait(false))
{
var row = ReadBucket(reader, family);
rows[row.Bucket] = row;
evaluated[row.Bucket] = AggregateEvaluator.Evaluate(aggregate, row, plan.BucketStart(row.Bucket), now);
}
}
var result = new List<DataValueSnapshot>((int)bucketCount);
for (long i = 0; i < bucketCount; i++)
{
var bucketStart = DateTime.SpecifyKind(new DateTime(start.Ticks + i * intervalTicks), DateTimeKind.Utc);
result.Add(rows.TryGetValue(i, out var row)
? Evaluate(aggregate, row, bucketStart, now)
: NoData(bucketStart, now));
}
if (reverse) result.Reverse();
var result = plan.Assemble(evaluated, bucketStart => AggregateEvaluator.NoData(bucketStart, now));
RecordSuccess();
return new HistoryReadResult(result, ContinuationPoint: null);
@@ -414,56 +351,13 @@ public sealed class SqlServerHistorianDataSource : IHistorianDataSource
}
}
private static DataValueSnapshot NoData(DateTime t, DateTime now) =>
new(null, HistorianInfoBits.WithCalculated(SqlHistorianStatusCodes.BadNoData), t, now);
private static DataValueSnapshot Evaluate(HistoryAggregateType aggregate, BucketRow r, DateTime bucketStart, DateTime now)
{
DataValueSnapshot Calc(object? value, uint status, DateTime? t = null) =>
new(value, HistorianInfoBits.WithCalculated(status), t ?? bucketStart, now);
if (r.Total == 0) return NoData(bucketStart, now);
switch (aggregate)
{
case HistoryAggregateType.PercentGood: return Calc(100d * r.Good / r.Total, SqlHistorianStatusCodes.Good);
case HistoryAggregateType.PercentBad: return Calc(100d * r.Bad / r.Total, SqlHistorianStatusCodes.Good);
}
if (r.Usable == 0) return NoData(bucketStart, now);
var anyText = aggregate is HistoryAggregateType.Count or HistoryAggregateType.Start or HistoryAggregateType.End;
if (!anyText && r.NonNumeric > 0)
return Calc(null, SqlHistorianStatusCodes.BadAggregateInvalidInputs);
var status = r.Good == r.Total ? SqlHistorianStatusCodes.Good : SqlHistorianStatusCodes.UncertainDataSubNormal;
switch (aggregate)
{
case HistoryAggregateType.Count: return Calc(r.Usable, status);
case HistoryAggregateType.Average: return Calc(r.Avg, status);
case HistoryAggregateType.Minimum: return Calc(r.Min, status);
case HistoryAggregateType.Maximum: return Calc(r.Max, status);
case HistoryAggregateType.Total: return Calc(r.Sum, status);
case HistoryAggregateType.Range: return Calc(r.Max - r.Min, status);
case HistoryAggregateType.Start: return Calc(r.First.ToObject(), status);
case HistoryAggregateType.End: return Calc(r.Last.ToObject(), status);
case HistoryAggregateType.Delta:
return r.First.TryGetDouble(out var a) && r.Last.TryGetDouble(out var b)
? Calc(b - a, status)
: Calc(null, SqlHistorianStatusCodes.BadAggregateInvalidInputs);
case HistoryAggregateType.MinimumActualTime:
case HistoryAggregateType.MaximumActualTime:
return Calc(r.Extreme, status, r.ExtremeTime);
default:
throw new HistoryAggregateNotSupportedException(aggregate);
}
}
private static BucketRow ReadBucket(SqlDataReader reader, SqlAggregateFamily family)
private static BucketRow ReadBucket(SqlDataReader reader, AggregateFamily family)
{
static double? D(SqlDataReader r, int i) => r.IsDBNull(i) ? null : r.GetDouble(i);
static long? L(SqlDataReader r, int i) => r.IsDBNull(i) ? null : r.GetInt64(i);
static string? S(SqlDataReader r, int i) => r.IsDBNull(i) ? null : r.GetString(i);
static HistorianValue V(SqlDataReader r, int i) =>
r.IsDBNull(i) ? HistorianValue.Empty : SampleRowMapper.FromColumns(r.GetByte(i), D(r, i + 1), L(r, i + 2), S(r, i + 3));
r.IsDBNull(i) ? HistorianValue.Empty : SampleColumnMapper.FromColumns(r.GetByte(i), D(r, i + 1), L(r, i + 2), S(r, i + 3));
var row = new BucketRow
{
@@ -477,13 +371,13 @@ public sealed class SqlServerHistorianDataSource : IHistorianDataSource
var f = SqlAggregateQueries.FirstFamilyColumn;
switch (family)
{
case SqlAggregateFamily.Simple:
case AggregateFamily.Simple:
row.Avg = D(reader, f);
row.Min = D(reader, f + 1);
row.Max = D(reader, f + 2);
row.Sum = D(reader, f + 3);
break;
case SqlAggregateFamily.StartEnd:
case AggregateFamily.StartEnd:
row.First = V(reader, f);
row.Last = V(reader, f + 5);
break;
@@ -495,44 +389,23 @@ public sealed class SqlServerHistorianDataSource : IHistorianDataSource
return row;
}
/// <summary>One aggregate query row (one non-empty bucket).</summary>
private sealed class BucketRow
{
public long Bucket;
public int Total, Usable, Good, Bad, NonNumeric;
public double? Avg, Min, Max, Sum, Extreme;
public DateTime? ExtremeTime;
public HistorianValue First, Last;
}
// ------------------------------------------------------------------------------------------ Shared
/// <summary>A stored sample read from the <c>{SampleColumns}</c> column run starting at an ordinal.</summary>
private readonly record struct StoredSample(DateTime TimeUtc, HistorianValue Value, uint Status)
{
/// <summary>Severity Good or Uncertain, and a value present (an Empty sample is an outage marker).</summary>
public bool IsUsable => !StatusBits.IsBad(Status) && Value.Kind != HistorianValueKind.Empty;
/// <summary>Usable with severity Good.</summary>
public bool IsGood => IsUsable && StatusBits.IsGood(Status);
}
/// <summary>
/// Reads <c>SourceTimeUtc, ValueKind, NumValue, IntValue, StrValue, StatusCode</c> starting at <paramref name="o"/>.
/// An Empty (no-value) row whose stored status is not Bad is clamped to <c>BadNoData</c> — defence in depth for
/// rows written before the writer forced every Empty sample Bad.
/// An Empty (no-value) row whose stored status is not Bad is clamped to <c>BadNoData</c>
/// (<see cref="StoredSample.FromStored"/>) — defence in depth for rows written before the writer forced every
/// Empty sample Bad.
/// </summary>
private static StoredSample ReadStored(SqlDataReader r, int o)
{
var value = SampleRowMapper.FromColumns(
var value = SampleColumnMapper.FromColumns(
r.GetByte(o + 1),
r.IsDBNull(o + 2) ? null : r.GetDouble(o + 2),
r.IsDBNull(o + 3) ? null : r.GetInt64(o + 3),
r.IsDBNull(o + 4) ? null : r.GetString(o + 4));
var status = SampleRowMapper.FromDbStatus(r.GetInt32(o + 5));
if (value.Kind == HistorianValueKind.Empty && !StatusBits.IsBad(status))
status = SqlHistorianStatusCodes.BadNoData;
return new StoredSample(SqlHistorianConnection.GetUtc(r, o), value, status);
var status = SampleColumnMapper.FromDbStatus(r.GetInt32(o + 5));
return StoredSample.FromStored(SqlHistorianConnection.GetUtc(r, o), value, status);
}
/// <summary>Read-only tag lookup (never creates a row); runs the schema bootstrap on a cache miss.</summary>
@@ -541,50 +414,15 @@ public sealed class SqlServerHistorianDataSource : IHistorianDataSource
// ------------------------------------------------------------------------------------------ Health
/// <inheritdoc />
public HistorianHealthSnapshot GetHealthSnapshot()
{
lock (_healthLock)
{
var open = _totalSuccesses > 0 && _consecutiveFailures == 0;
return new HistorianHealthSnapshot(
TotalQueries: _totalQueries,
TotalSuccesses: _totalSuccesses,
TotalFailures: _totalFailures,
ConsecutiveFailures: _consecutiveFailures,
LastSuccessTime: _lastSuccessUtc,
LastFailureTime: _lastFailureUtc,
LastError: _lastError,
ProcessConnectionOpen: open,
EventConnectionOpen: open,
ActiveProcessNode: null,
ActiveEventNode: null,
Nodes: []);
}
}
public HistorianHealthSnapshot GetHealthSnapshot() => _health.Snapshot();
private void RecordSuccess()
{
lock (_healthLock)
{
_totalQueries++;
_totalSuccesses++;
_consecutiveFailures = 0;
_lastSuccessUtc = DateTime.UtcNow;
}
}
private void RecordSuccess() => _health.RecordSuccess();
private void RecordFailure(Exception ex)
{
// A cancelled read (deadline / client abort) is not a backend fault.
// A cancelled read (deadline / client abort) is not a backend fault (the counters ignore it too).
if (ex is OperationCanceledException) return;
lock (_healthLock)
{
_totalQueries++;
_totalFailures++;
_consecutiveFailures++;
_lastFailureUtc = DateTime.UtcNow;
_lastError = $"{ex.GetType().Name}: {ex.Message}";
}
_health.RecordFailure(ex);
_log.LogDebug("SQL historian read failed ({ExceptionType})", ex.GetType().Name);
}
src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerHistorianDataUpdater.cs
+29 -86
View File
@@ -1,18 +1,9 @@
using Microsoft.Data.SqlClient;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer;
/// <summary>Status codes owned by the updater (named consts so <c>StatusCodeParityTests</c> sees them).</summary>
internal static class SqlHistorianUpdateStatusCodes
{
/// <summary>
/// A value that cannot be stored: a Double NaN / ±Infinity, an Empty (null) value with a non-Bad status, or a
/// tag name longer than <see cref="SqlServerTagCatalog.MaxNameLength"/>.
/// </summary>
public const uint BadInvalidArgument = 0x80AB0000u;
}
/// <summary>
/// OPC UA HistoryUpdate against the SQL Server sample table (spec §4 "Updates"). One transaction per call;
/// exactly one status per input item, in input order.
@@ -26,7 +17,7 @@ internal static class SqlHistorianUpdateStatusCodes
/// accepted with a Bad status (an operator-authored outage marker); with a Good or Uncertain status it is
/// <c>BadInvalidArgument</c>, because a stored sample is never served better than "no value". Other items
/// still proceed.</para>
/// <para><b>Tag names over <see cref="SqlServerTagCatalog.MaxNameLength"/> characters</b> are refused, never
/// <para><b>Tag names over <see cref="HistorianTagNames.MaxLength"/> characters</b> are refused, never
/// truncated: every item of the call (and DeleteRaw's single result) reports <c>BadInvalidArgument</c> and nothing
/// is written.</para>
/// <para><b>Unknown tag.</b> Insert/Update create it via <see cref="SqlServerTagCatalog"/>; Replace and the
@@ -52,70 +43,26 @@ internal sealed class SqlServerHistorianDataUpdater : IHistorianDataUpdater
{
ArgumentNullException.ThrowIfNull(tag);
ArgumentNullException.ThrowIfNull(values);
var results = new uint[values.Count];
if (!SqlServerTagCatalog.IsValidName(tag))
{
Array.Fill(results, SqlHistorianUpdateStatusCodes.BadInvalidArgument);
return results;
}
var winner = new Dictionary<long, int>(); // tick -> index of the item actually sent
for (var i = 0; i < values.Count; i++)
{
var v = values[i];
if (v.TimestampUtc is not { } ts || ts == DateTime.MinValue)
{
results[i] = SqlHistorianStatusCodes.BadInvalidTimestamp;
continue;
}
if ((v.Value.Kind == HistorianValueKind.Double && !double.IsFinite(v.Value.Number))
|| (v.Value.Kind == HistorianValueKind.Empty && !StatusBits.IsBad(v.StatusCode)))
{
results[i] = SqlHistorianUpdateStatusCodes.BadInvalidArgument;
continue;
}
if (!winner.TryGetValue(ts.Ticks, out _))
winner[ts.Ticks] = i;
else if (mode == HistoryUpdateMode.Insert)
results[i] = SqlHistorianStatusCodes.BadEntryExists; // first occurrence stays the winner
else
winner[ts.Ticks] = i; // last wins
}
var send = winner.Values.OrderBy(i => i).ToList();
if (send.Count == 0) return results;
var plan = HistoryUpdatePlanner.Plan(tag, mode, values);
if (plan.Winners.Count == 0) return HistoryUpdatePlanner.Complete(plan, mode, new Dictionary<long, UpsertOutcome>());
var tagId = mode == HistoryUpdateMode.Replace
? await _tags.TryGetTagIdAsync(tag, ct).ConfigureAwait(false)
: await _tags.GetOrCreateTagIdAsync(tag, SqlHistorianDataTypes.FromKind(values[send[0]].Value.Kind), ct).ConfigureAwait(false);
: await _tags.GetOrCreateTagIdAsync(tag, HistorianDataTypes.FromKind(plan.Winners[0].Row.Value.Kind), ct).ConfigureAwait(false);
var outcome = tagId is null
? new Dictionary<long, uint>()
: await MergeAsync(tagId.Value, mode, send.Select(i => new SqlSampleRow(
values[i].TimestampUtc!.Value, values[i].Value, values[i].StatusCode)).ToList(), ct).ConfigureAwait(false);
// No OUTPUT row means the MERGE's absent branch: Insert => matched (exists); Replace => not matched.
var absent = mode == HistoryUpdateMode.Replace
? SqlHistorianStatusCodes.BadNoEntryExists
: SqlHistorianStatusCodes.BadEntryExists;
foreach (var i in send)
results[i] = outcome.TryGetValue(values[i].TimestampUtc!.Value.Ticks, out var code) ? code : absent;
// Superseded earlier duplicates (Replace/Update) share the winner's code.
if (mode != HistoryUpdateMode.Insert)
for (var i = 0; i < values.Count; i++)
if (values[i].TimestampUtc is { } ts && results[i] == 0
&& winner.TryGetValue(ts.Ticks, out var w) && w != i)
results[i] = results[w];
return results;
? new Dictionary<long, UpsertOutcome>()
: await MergeAsync(tagId.Value, mode, plan.Winners.Select(w => w.Row).ToList(), ct).ConfigureAwait(false);
return HistoryUpdatePlanner.Complete(plan, mode, outcome);
}
/// <inheritdoc/>
public async Task<uint> DeleteRawAsync(string tag, DateTime startUtc, DateTime endUtc, CancellationToken ct)
{
ArgumentNullException.ThrowIfNull(tag);
if (!SqlServerTagCatalog.IsValidName(tag)) return SqlHistorianUpdateStatusCodes.BadInvalidArgument;
if (!HistorianTagNames.IsValid(tag)) return HistorianStatusCodes.BadInvalidArgument;
var tagId = await _tags.TryGetTagIdAsync(tag, ct).ConfigureAwait(false);
if (tagId is null) return SqlHistorianStatusCodes.BadNoData;
if (tagId is null) return HistoryUpdatePlanner.DeleteRawResult(tagKnown: false, 0);
await using var conn = await _conn.OpenAsync(ct).ConfigureAwait(false);
await using var tx = (SqlTransaction)await conn.BeginTransactionAsync(ct).ConfigureAwait(false);
int rows;
@@ -128,7 +75,7 @@ internal sealed class SqlServerHistorianDataUpdater : IHistorianDataUpdater
rows = await cmd.ExecuteNonQueryAsync(ct).ConfigureAwait(false);
}
await tx.CommitAsync(ct).ConfigureAwait(false);
return rows > 0 ? SqlHistorianStatusCodes.Good : SqlHistorianStatusCodes.BadNoData;
return HistoryUpdatePlanner.DeleteRawResult(tagKnown: true, rows);
}
/// <inheritdoc/>
@@ -136,22 +83,19 @@ internal sealed class SqlServerHistorianDataUpdater : IHistorianDataUpdater
{
ArgumentNullException.ThrowIfNull(tag);
ArgumentNullException.ThrowIfNull(timesUtc);
var results = new uint[timesUtc.Count];
if (!SqlServerTagCatalog.IsValidName(tag))
if (!HistorianTagNames.IsValid(tag))
{
Array.Fill(results, SqlHistorianUpdateStatusCodes.BadInvalidArgument);
return results;
var refused = new uint[timesUtc.Count];
Array.Fill(refused, HistorianStatusCodes.BadInvalidArgument);
return refused;
}
var first = new Dictionary<long, int>();
for (var i = 0; i < timesUtc.Count; i++)
{
results[i] = SqlHistorianStatusCodes.BadNoEntryExists;
if (timesUtc[i] == DateTime.MinValue) results[i] = SqlHistorianStatusCodes.BadInvalidTimestamp;
else first.TryAdd(timesUtc[i].Ticks, i);
}
if (first.Count == 0) return results;
var distinct = new HashSet<long>();
foreach (var t in timesUtc)
if (t != DateTime.MinValue) distinct.Add(t.Ticks);
var deleted = new HashSet<long>();
if (distinct.Count == 0) return HistoryUpdatePlanner.DeleteAtTimeResults(timesUtc, deleted, tagKnown: false);
var tagId = await _tags.TryGetTagIdAsync(tag, ct).ConfigureAwait(false);
if (tagId is null) return results;
if (tagId is null) return HistoryUpdatePlanner.DeleteAtTimeResults(timesUtc, deleted, tagKnown: false);
await using var conn = await _conn.OpenAsync(ct).ConfigureAwait(false);
await using var tx = (SqlTransaction)await conn.BeginTransactionAsync(ct).ConfigureAwait(false);
@@ -160,18 +104,17 @@ internal sealed class SqlServerHistorianDataUpdater : IHistorianDataUpdater
{
cmd.Parameters.Add(new SqlParameter("@tagId", System.Data.SqlDbType.Int) { Value = tagId.Value });
_conn.AddStructured(cmd, "@times", "TimeRow",
SampleRowMapper.ToTimeRecords(first.Keys.Select(t => new DateTime(t, DateTimeKind.Utc))));
SampleRowMapper.ToTimeRecords(distinct.Select(t => new DateTime(t, DateTimeKind.Utc))));
await using var rd = await cmd.ExecuteReaderAsync(ct).ConfigureAwait(false);
while (await rd.ReadAsync(ct).ConfigureAwait(false))
if (first.TryGetValue(SqlHistorianConnection.GetUtc(rd, 0).Ticks, out var idx))
results[idx] = SqlHistorianStatusCodes.Good;
deleted.Add(SqlHistorianConnection.GetUtc(rd, 0).Ticks);
}
await tx.CommitAsync(ct).ConfigureAwait(false);
return results;
return HistoryUpdatePlanner.DeleteAtTimeResults(timesUtc, deleted, tagKnown: true);
}
private async Task<Dictionary<long, uint>> MergeAsync(int tagId, HistoryUpdateMode mode,
List<SqlSampleRow> rows, CancellationToken ct)
private async Task<Dictionary<long, UpsertOutcome>> MergeAsync(int tagId, HistoryUpdateMode mode,
List<HistorianSampleRow> rows, CancellationToken ct)
{
var sql = $"MERGE {_conn.Q("Sample")} WITH (HOLDLOCK) AS t USING @rows AS r ON t.TagId = @tagId AND t.SourceTimeUtc = r.SourceTimeUtc ";
if (mode != HistoryUpdateMode.Insert)
@@ -182,7 +125,7 @@ internal sealed class SqlServerHistorianDataUpdater : IHistorianDataUpdater
"VALUES (@tagId, r.SourceTimeUtc, r.ValueKind, r.NumValue, r.IntValue, r.StrValue, r.StatusCode, SYSUTCDATETIME()) ";
sql += "OUTPUT $action, r.SourceTimeUtc;";
var outcome = new Dictionary<long, uint>();
var outcome = new Dictionary<long, UpsertOutcome>();
await using var conn = await _conn.OpenAsync(ct).ConfigureAwait(false);
await using var tx = (SqlTransaction)await conn.BeginTransactionAsync(ct).ConfigureAwait(false);
await using (var cmd = _conn.CreateCommand(conn, sql, tx))
@@ -192,8 +135,8 @@ internal sealed class SqlServerHistorianDataUpdater : IHistorianDataUpdater
await using var rd = await cmd.ExecuteReaderAsync(ct).ConfigureAwait(false);
while (await rd.ReadAsync(ct).ConfigureAwait(false))
outcome[SqlHistorianConnection.GetUtc(rd, 1).Ticks] = rd.GetString(0) == "INSERT"
? SqlHistorianStatusCodes.GoodEntryInserted
: SqlHistorianStatusCodes.GoodEntryReplaced;
? UpsertOutcome.Inserted
: UpsertOutcome.Replaced;
}
await tx.CommitAsync(ct).ConfigureAwait(false);
return outcome;
src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerHistorianOptions.cs
-8
View File
@@ -44,11 +44,3 @@ public sealed partial class SqlServerHistorianOptions
return errors;
}
}
/// <summary>Thrown when <c>AutoCreateSchema=false</c> and the historian schema is absent or behind.</summary>
public sealed class HistorianSchemaMissingException : Exception
{
/// <summary>Creates the exception.</summary>
/// <param name="message">Message naming the missing version.</param>
public HistorianSchemaMissingException(string message) : base(message) { }
}
src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerHistorianValueWriter.cs
+5 -37
View File
@@ -2,6 +2,7 @@ using Microsoft.Data.SqlClient;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Logging.Abstractions;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer;
@@ -21,6 +22,7 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer;
/// <b>Tag names over <see cref="SqlServerTagCatalog.MaxNameLength"/> characters</b> are refused (the write
/// returns false, a per-tag failure for the recorder's #538 quarantine), never truncated onto another tag.
/// <b>Duplicate timestamps within a batch</b> collapse to the first (matching insert-if-absent across batches).
/// The batch normalisation itself is <see cref="LiveWritePreparer.Prepare"/>.
/// </remarks>
internal sealed class SqlServerHistorianValueWriter : IHistorianValueWriter
{
@@ -51,7 +53,7 @@ internal sealed class SqlServerHistorianValueWriter : IHistorianValueWriter
{
try
{
if (!SqlServerTagCatalog.IsValidName(tag))
if (!HistorianTagNames.IsValid(tag))
{
_log.LogWarning("SQL historian live-value write refused: tag name is missing or longer than {Max} characters.",
SqlServerTagCatalog.MaxNameLength);
@@ -60,8 +62,8 @@ internal sealed class SqlServerHistorianValueWriter : IHistorianValueWriter
await _bootstrap.EnsureAsync(ct).ConfigureAwait(false);
if (values is null || values.Count == 0) return true;
var rows = Prepare(values, _utcNow);
var dataType = SqlHistorianDataTypes.FromKind(rows.Select(r => r.Value.Kind).FirstOrDefault(k => k != HistorianValueKind.Empty));
var rows = LiveWritePreparer.Prepare(values, _utcNow);
var dataType = HistorianDataTypes.FromKind(rows.Select(r => r.Value.Kind).FirstOrDefault(k => k != HistorianValueKind.Empty));
var tagId = await _catalog.GetOrCreateTagIdAsync(tag, dataType, ct).ConfigureAwait(false);
var sql =
@@ -94,38 +96,4 @@ internal sealed class SqlServerHistorianValueWriter : IHistorianValueWriter
return false;
}
}
/// <summary>
/// Normalises a batch for the TVP: null timestamp ⇒ now, non-finite doubles ⇒ Empty, every Empty forced Bad
/// (non-finite ⇒ <c>BadOutOfRange</c>, otherwise <c>BadNoData</c>, an existing Bad code kept), and duplicate
/// timestamps collapsed to the first.
/// </summary>
/// <param name="values">Input samples, in append order.</param>
/// <param name="utcNow">Clock for null timestamps.</param>
/// <returns>Time-unique rows.</returns>
internal static List<SqlSampleRow> Prepare(IReadOnlyList<HistorizationValue> values, Func<DateTime> utcNow)
{
var seen = new HashSet<long>();
var rows = new List<SqlSampleRow>(values.Count);
foreach (var v in values)
{
var t = v.TimestampUtc ?? utcNow();
t = t.Kind == DateTimeKind.Local ? t.ToUniversalTime() : DateTime.SpecifyKind(t, DateTimeKind.Utc);
if (!seen.Add(t.Ticks)) continue;
var value = v.Value;
var status = v.StatusCode;
if (value.Kind == HistorianValueKind.Double && !double.IsFinite(value.Number))
{
value = HistorianValue.Empty;
if (!StatusBits.IsBad(status)) status = SqlHistorianStatusCodes.BadOutOfRange;
}
else if (value.Kind == HistorianValueKind.Empty && !StatusBits.IsBad(status))
{
status = SqlHistorianStatusCodes.BadNoData;
}
rows.Add(new SqlSampleRow(t, value, status));
}
return rows;
}
}
src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerSchemaBootstrapper.cs
+8 -14
View File
@@ -1,17 +1,17 @@
using System.Data;
using System.Reflection;
using System.Runtime.ExceptionServices;
using System.Text.RegularExpressions;
using Microsoft.Data.SqlClient;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Logging.Abstractions;
using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer;
/// <summary>
/// Lazily applies the embedded, idempotent schema scripts once per instance, under a SQL application lock
/// so concurrent nodes/instances serialise. A failure is NOT latched: an <see cref="EnsureAsync"/> after
/// <see cref="FailureCacheDuration"/> retries. Within that window the last failure is re-thrown without touching
/// <see cref="BootstrapFailureCache.DefaultDuration"/> retries (<see cref="BootstrapFailureCache"/>). Within that window the last failure is re-thrown without touching
/// SQL, so a missing schema (or a SQL outage) does not serialise every write, read and alarm event behind one
/// semaphore and one round-trip each.
/// With <c>AutoCreateSchema=false</c> it only verifies <c>MAX(Version) &gt;= 1</c>.
@@ -24,17 +24,12 @@ internal sealed partial class SqlServerSchemaBootstrapper
private const string LockResource = "OtOpcUaHistorianSchema";
private const int LockTimeoutMs = 30_000;
/// <summary>How long the last bootstrap failure is re-thrown before SQL is tried again.</summary>
public static readonly TimeSpan FailureCacheDuration = TimeSpan.FromSeconds(5);
private readonly SqlHistorianConnection _conn;
private readonly bool _autoCreate;
private readonly ILogger _log;
private readonly TimeProvider _time;
private readonly SemaphoreSlim _gate = new(1, 1);
private readonly BootstrapFailureCache _failure; // guarded by _gate
private volatile bool _ready;
private ExceptionDispatchInfo? _lastFailure; // guarded by _gate
private DateTimeOffset _retryAfter; // guarded by _gate
/// <summary>Creates the bootstrapper.</summary>
/// <param name="conn">Connection helper.</param>
@@ -47,7 +42,7 @@ internal sealed partial class SqlServerSchemaBootstrapper
_conn = conn;
_autoCreate = options.AutoCreateSchema;
_log = log ?? NullLogger.Instance;
_time = timeProvider ?? TimeProvider.System;
_failure = new BootstrapFailureCache(timeProvider ?? TimeProvider.System, BootstrapFailureCache.DefaultDuration);
}
/// <summary>Ensures the schema is ready; no-op after the first success.</summary>
@@ -61,23 +56,22 @@ internal sealed partial class SqlServerSchemaBootstrapper
try
{
if (_ready) return;
if (_lastFailure is { } cached && _time.GetUtcNow() < _retryAfter) cached.Throw();
_failure.ThrowIfCached();
try
{
await using var conn = await _conn.OpenAsync(ct).ConfigureAwait(false);
if (_autoCreate) await ApplyAsync(conn, ct).ConfigureAwait(false);
else await VerifyAsync(conn, ct).ConfigureAwait(false);
_ready = true;
_lastFailure = null;
_failure.Clear();
}
catch (Exception ex) when (ex is not OperationCanceledException || !ct.IsCancellationRequested)
{
// A caller's own cancellation is not a backend failure and is not cached.
_lastFailure = ExceptionDispatchInfo.Capture(ex);
_retryAfter = _time.GetUtcNow() + FailureCacheDuration;
_failure.Record(ex);
_log.LogWarning("Historian schema bootstrap for [{Schema}] failed ({Exception}: {Message}); " +
"historian calls fail fast for {Seconds} s before retrying.",
_conn.Schema, ex.GetType().Name, ex.Message, FailureCacheDuration.TotalSeconds);
_conn.Schema, ex.GetType().Name, ex.Message, BootstrapFailureCache.DefaultDuration.TotalSeconds);
throw;
}
}
src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerTagCatalog.cs
+5 -9
View File
@@ -1,5 +1,6 @@
using System.Collections.Concurrent;
using Microsoft.Data.SqlClient;
using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer;
@@ -13,8 +14,8 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer;
/// </summary>
internal sealed class SqlServerTagCatalog
{
/// <summary>Width of the <c>Tag.Name</c> column; longer names are refused.</summary>
public const int MaxNameLength = 400;
/// <summary>Width of the <c>Tag.Name</c> column (= <see cref="HistorianTagNames.MaxLength"/>); longer names are refused.</summary>
public const int MaxNameLength = HistorianTagNames.MaxLength;
private readonly SqlHistorianConnection _conn;
private readonly SqlServerSchemaBootstrapper _bootstrap;
@@ -29,18 +30,13 @@ internal sealed class SqlServerTagCatalog
_bootstrap = bootstrap;
}
/// <summary>True when <paramref name="name"/> fits the <c>Tag.Name</c> column without truncation.</summary>
/// <param name="name">Historian tag name.</param>
/// <returns>Whether the name is non-null and at most <see cref="MaxNameLength"/> characters.</returns>
public static bool IsValidName(string? name) => name is not null && name.Length <= MaxNameLength;
/// <summary>Read-only lookup; never creates a row. A name over <see cref="MaxNameLength"/> is not found.</summary>
/// <param name="name">Historian tag name.</param>
/// <param name="ct">Cancellation.</param>
/// <returns>The TagId, or null when the tag does not exist.</returns>
public async Task<int?> TryGetTagIdAsync(string name, CancellationToken ct)
{
if (!IsValidName(name)) return null;
if (!HistorianTagNames.IsValid(name)) return null;
if (_cache.TryGetValue(name, out var cached)) return cached;
await _bootstrap.EnsureAsync(ct).ConfigureAwait(false);
await using var conn = await _conn.OpenAsync(ct).ConfigureAwait(false);
@@ -57,7 +53,7 @@ internal sealed class SqlServerTagCatalog
/// <exception cref="ArgumentException">The name is longer than <see cref="MaxNameLength"/>.</exception>
public async Task<int> GetOrCreateTagIdAsync(string name, byte dataType, CancellationToken ct)
{
if (!IsValidName(name))
if (!HistorianTagNames.IsValid(name))
throw new ArgumentException($"Historian tag name exceeds {MaxNameLength} characters.", nameof(name));
if (_cache.TryGetValue(name, out var cached)) return cached;
await _bootstrap.EnsureAsync(ct).ConfigureAwait(false);
src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.csproj
+2
View File
@@ -11,6 +11,8 @@
<ItemGroup>
<ProjectReference Include="..\..\Core\ZB.MOM.WW.OtOpcUa.Core.Abstractions\ZB.MOM.WW.OtOpcUa.Core.Abstractions.csproj"/>
<ProjectReference Include="..\..\Core\ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian\ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian.csproj"/>
<!-- The database-independent historian core (status codes, evaluators, planners) shared with the QuestDB backend. -->
<ProjectReference Include="..\ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common\ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.csproj"/>
<!-- Runtime owns IHistorianBackendProvider / ServerHistorianOptions; same diamond as Driver.Historian.Gateway
(Host -> {Runtime, SqlServer}, SqlServer -> Runtime). This project does NOT reference the Gateway project. -->
<ProjectReference Include="..\..\Server\ZB.MOM.WW.OtOpcUa.Runtime\ZB.MOM.WW.OtOpcUa.Runtime.csproj"/>
src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/HistorianBackendRegistry.cs
+13
View File
@@ -1,6 +1,7 @@
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.Options;
using ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway;
using ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb;
using ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer;
using ZB.MOM.WW.OtOpcUa.Runtime.Historian;
@@ -25,11 +26,23 @@ public static class HistorianBackendRegistry
return o;
}
/// <summary>Backend name for the QuestDB historian.</summary>
public const string QuestDbBackendName = "QuestDb";
/// <summary>Binds the <c>ServerHistorian:QuestDb</c> section (shared by the registry and the validator).</summary>
public static QuestDbHistorianOptions BindQuestDb(IConfiguration configuration)
{
var o = new QuestDbHistorianOptions();
configuration.GetSection($"{ServerHistorianOptions.SectionName}:QuestDb").Bind(o);
return o;
}
private static readonly IReadOnlyDictionary<string, Func<ServerHistorianOptions, IConfiguration, IHistorianBackendProvider>> Factories =
new Dictionary<string, Func<ServerHistorianOptions, IConfiguration, IHistorianBackendProvider>>(StringComparer.OrdinalIgnoreCase)
{
[ServerHistorianOptions.GatewayBackendName] = (o, _) => new GatewayHistorianProvider(o),
[SqlServerBackendName] = (_, c) => new SqlServerHistorianProvider(BindSqlServer(c)),
[QuestDbBackendName] = (_, c) => new QuestDbHistorianProvider(BindQuestDb(c)),
};
/// <summary>The registered backend names, for validation messages.</summary>
src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/ServerHistorianOptionsValidator.cs
+7
View File
@@ -78,6 +78,13 @@ public sealed class ServerHistorianOptionsValidator : OptionsValidatorBase<Serve
return;
}
if (string.Equals(options.Backend?.Trim(), HistorianBackendRegistry.QuestDbBackendName, StringComparison.OrdinalIgnoreCase))
{
foreach (var error in HistorianBackendRegistry.BindQuestDb(_configuration).Validate())
builder.RequireThat(false, error);
return;
}
if (!options.IsGatewayBackend) return;
var endpointValid = Uri.TryCreate(options.Endpoint, UriKind.Absolute, out var uri)
src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs
+10 -6
View File
@@ -235,15 +235,19 @@ if (hasDriver)
// Backend selection: resolve the provider once (unknown name => OptionsValidationException naming the
// allowed values) and build every historian seam through it.
var historianProvider = HistorianBackendRegistry.Resolve(serverHistorianOptions, builder.Configuration);
builder.Services.AddSingleton<IHistorianBackendProvider>(historianProvider);
// Registered through a factory (not as an instance) so the container owns it and disposes it at shutdown
// (the QuestDb provider's shared Npgsql pool); every seam below resolves the provider from the container
// rather than capturing the local, so whichever seam builds the backend's shared state also makes the
// container track the provider for disposal.
builder.Services.AddSingleton<IHistorianBackendProvider>(_ => historianProvider);
// HistoryUpdate is gated like HistoryRead: None unconditionally, the real holder only when enabled.
builder.Services.AddSingleton(HistorianUpdaterHolder.None);
if (serverHistorianOptions.Enabled)
builder.Services.AddSingleton(sp => new HistorianUpdaterHolder(historianProvider.CreateUpdater(sp)));
builder.Services.AddSingleton(sp => new HistorianUpdaterHolder(sp.GetRequiredService<IHistorianBackendProvider>().CreateUpdater(sp)));
builder.Services.AddAlarmHistorian(
builder.Configuration,
(_, sp) => historianProvider.CreateAlarmWriter(sp));
(_, sp) => sp.GetRequiredService<IHistorianBackendProvider>().CreateAlarmWriter(sp));
// Node-local LocalDb: caches the deployed-configuration artifact so this node can boot from
// its last-known-good config when central SQL is unreachable, and optionally replicates that
@@ -269,7 +273,7 @@ if (hasDriver)
// node manager's HistoryRead overrides block-bridge to. Runtime owns the gating.
builder.Services.AddServerHistorian(
builder.Configuration,
(_, sp) => historianProvider.CreateDataSource(sp));
(_, sp) => sp.GetRequiredService<IHistorianBackendProvider>().CreateDataSource(sp));
// Config-gated historian tag provisioning. When the ServerHistorian section is enabled this registers the
// selected backend's provisioner (GatewayTagProvisioner -> the gateway's EnsureTags, or the SqlServer Tag-row
@@ -280,7 +284,7 @@ if (hasDriver)
// the SAME ServerHistorian:Enabled flag as the read path — both use the one resolved backend.
builder.Services.AddHistorianProvisioning(
builder.Configuration,
(_, sp) => historianProvider.CreateProvisioning(sp));
(_, sp) => sp.GetRequiredService<IHistorianBackendProvider>().CreateProvisioning(sp));
// Continuous historization of driver (non-Galaxy) tag values. Gated on ContinuousHistorization:Enabled
// AND ServerHistorian:Enabled: the recorder drains driver-tag live values through the selected backend's
@@ -319,7 +323,7 @@ if (hasDriver)
// The backend's live-value writer. On Gateway it opens its OWN gRPC channel to the same gateway (a second
// channel to a co-located sidecar is cheap — the gateway pools the historian sessions server-side); on
// SqlServer it shares the provider's connection helper, schema bootstrapper and tag catalog.
builder.Services.AddSingleton<IHistorianValueWriter>(sp => historianProvider.CreateValueWriter(sp));
builder.Services.AddSingleton<IHistorianValueWriter>(sp => sp.GetRequiredService<IHistorianBackendProvider>().CreateValueWriter(sp));
}
// Bind every cross-platform driver factory before AddAkka resolves IDriverFactory — replaces
src/Server/ZB.MOM.WW.OtOpcUa.Host/ZB.MOM.WW.OtOpcUa.Host.csproj
+1
View File
@@ -81,6 +81,7 @@
<ProjectReference Include="..\..\Drivers\ZB.MOM.WW.OtOpcUa.Driver.Galaxy\ZB.MOM.WW.OtOpcUa.Driver.Galaxy.csproj"/>
<ProjectReference Include="..\..\Drivers\ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway\ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.csproj"/>
<ProjectReference Include="..\..\Drivers\ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer\ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.csproj"/>
<ProjectReference Include="..\..\Drivers\ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb\ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.csproj"/>
<ProjectReference Include="..\..\Drivers\ZB.MOM.WW.OtOpcUa.Driver.Modbus\ZB.MOM.WW.OtOpcUa.Driver.Modbus.csproj"/>
<ProjectReference Include="..\..\Drivers\ZB.MOM.WW.OtOpcUa.Driver.Mqtt\ZB.MOM.WW.OtOpcUa.Driver.Mqtt.csproj"/>
<ProjectReference Include="..\..\Drivers\ZB.MOM.WW.OtOpcUa.Driver.MTConnect\ZB.MOM.WW.OtOpcUa.Driver.MTConnect.csproj"/>
src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/HistoryPaging.cs
+14 -5
View File
@@ -212,25 +212,34 @@ internal static class HistoryPaging
/// after <paramref name="endUtc"/> (the reverse window excludes <c>end</c>; a resume at exactly <c>end</c>
/// would turn into a <c>start == end</c> point read and return the excluded sample). Forward is unchanged.
/// </param>
/// <param name="resolutionTicks">
/// The backend's storage timestamp resolution (<c>IHistorianDataSource.TimestampResolutionTicks</c>, &gt;= 1). A
/// drained cluster steps off <paramref name="boundaryT"/> by this many ticks rather than by one: a backend that
/// truncates its inputs to its resolution (QuestDB, 10 ticks) would otherwise truncate <c>T + 1 tick</c> back
/// to <c>T</c> and return the drained sample again, forever. Lossless for the same reason one tick is at tick
/// precision: no stored timestamp lies strictly between <c>T</c> and <c>T ± resolution</c>. The default
/// <c>1</c> keeps the tick-precision behaviour exactly.
/// </param>
public static void SliceTieCluster(
int clusterCount, int skip, uint cap, DateTime boundaryT, DateTime endUtc,
out int sliceStart, out int sliceCount, out DateTime? nextStartUtc, out int nextSkip,
bool reverse = false)
bool reverse = false, long resolutionTicks = 1)
{
ArgumentOutOfRangeException.ThrowIfLessThan(resolutionTicks, 1L);
sliceStart = Math.Min(skip, clusterCount);
sliceCount = Math.Min((int)cap, clusterCount - sliceStart);
var emitted = sliceStart + sliceCount;
if (emitted < clusterCount) { nextStartUtc = boundaryT; nextSkip = emitted; }
else if (reverse)
{
// DateTime.MinValue cannot step down; nothing can lie below it anyway.
var more = boundaryT > DateTime.MinValue && boundaryT.AddTicks(-1) > endUtc;
nextStartUtc = more ? boundaryT.AddTicks(-1) : null;
// Nothing can lie below DateTime.MinValue, so a boundary within one resolution step of it cannot step down.
var more = boundaryT.Ticks >= resolutionTicks && boundaryT.AddTicks(-resolutionTicks) > endUtc;
nextStartUtc = more ? boundaryT.AddTicks(-resolutionTicks) : null;
nextSkip = 0;
}
else
{
var next = boundaryT.AddTicks(1);
var next = boundaryT.AddTicks(resolutionTicks);
if (next <= endUtc) { nextStartUtc = next; nextSkip = 0; }
else { nextStartUtc = null; nextSkip = 0; }
}
src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OtOpcUaNodeManager.cs
+1 -1
View File
@@ -5931,7 +5931,7 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
HistoryPaging.SliceTieCluster(
cluster.Count, boundarySkip, numValuesPerNode, startUtc, endUtc,
out var sliceStart, out var sliceCount, out var nextStartUtc, out var nextSkip,
reverse: reverse);
reverse: reverse, resolutionTicks: Math.Max(1L, HistorianDataSource.TimestampResolutionTicks));
var slice = new List<DataValueSnapshot>(sliceCount);
for (var i = sliceStart; i < sliceStart + sliceCount; i++) slice.Add(cluster[i]);
tests/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions.Tests/ZB.MOM.WW.OtOpcUa.Core.Abstractions.Tests.csproj
+3
View File
@@ -43,6 +43,9 @@
DriverTypeNames guard skips it — but it does hard-code OPC UA status constants, which
puts it in StatusCodeParityTests' scope. -->
<ProjectReference Include="..\..\..\src\Drivers\ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway\ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.csproj"/>
<!-- The SQL historian backends' shared status constants (HistorianStatusCodes) live here. -->
<ProjectReference Include="..\..\..\src\Drivers\ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb\ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.csproj"/>
<ProjectReference Include="..\..\..\src\Drivers\ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common\ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.csproj"/>
<ProjectReference Include="..\..\..\src\Drivers\ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer\ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.csproj"/>
</ItemGroup>
tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/AtTimeReadTests.cs → tests/Drivers/Historian.Conformance/AtTimeConformance.cs
+48 -62
View File
@@ -2,18 +2,16 @@ using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Conformance;
/// <summary>
/// AtTime rules (spec §4, as amended after the T10 review): exact, linear, stepped, no-prior, and the two
/// Bad-neighbour cases — inside an outage (Bad previous) ⇒ null with the outage's own Bad code; before an
/// outage (usable previous, Bad next) ⇒ previous value held, Uncertain. Never bridges.
/// </summary>
[Collection(SqlHistorianCollection.Name)]
public sealed class AtTimeReadTests(SqlHistorianFixture fx)
public abstract class AtTimeConformance : ConformanceBase
{
private static readonly CancellationToken Ct = CancellationToken.None;
private static DateTime At(double s) => ReadTestDb.At(s);
// Hand-computed status literals: Interpolated = InfoType DataValue 0x400 | Interpolated 0x2.
private const uint GoodInterpolated = 0x00000402u;
@@ -33,13 +31,12 @@ public sealed class AtTimeReadTests(SqlHistorianFixture fx)
[Fact]
public async Task Exact_hit_returns_the_stored_value_and_status()
{
var db = await ReadTestDb.CreateAsync(fx);
await db.SeedAAsync();
var s = (await db.Source().ReadAtTimeAsync("A", [At(10), At(20)], Ct)).Samples;
await ConformanceData.SeedAAsync(B);
var s = (await B.CreateDataSource().ReadAtTimeAsync("A", [At(10), At(20)], Ct)).Samples;
s[0].Value.ShouldBe(20d);
s[0].StatusCode.ShouldBe(0u);
s[0].SourceTimestampUtc.ShouldBe(At(10));
s[0].ServerTimestampUtc.ShouldBe(At(11)); // the stored ServerTimeUtc
s[0].SourceTimestampUtc.ShouldBe(At(10)); // the requested time itself, never rounded (M3)
s[0].ServerTimestampUtc.ShouldBe(B.Round(At(11))); // the stored ServerTimeUtc
s[1].Value.ShouldBe(30d); // a stored Bad sample comes back as stored
s[1].StatusCode.ShouldBe(0x80000000u);
}
@@ -47,33 +44,31 @@ public sealed class AtTimeReadTests(SqlHistorianFixture fx)
[Fact]
public async Task Between_two_good_doubles_is_linearly_interpolated_good()
{
var db = await ReadTestDb.CreateAsync(fx);
await db.SeedAAsync();
var s = (await db.Source().ReadAtTimeAsync("A", [At(5), At(2.5), At(32.5)], Ct)).Samples;
await ConformanceData.SeedAAsync(B);
var s = (await B.CreateDataSource().ReadAtTimeAsync("A", [At(5), At(2.5), At(32.5)], Ct)).Samples;
s[0].Value.ShouldBe(15d); // 10 + (20-10) * 5/10
s[1].Value.ShouldBe(12.5d); // 10 + 10 * 2.5/10
s[2].Value.ShouldBe(42.5d); // 40 + 10 * 2.5/10
s.ShouldAllBe(x => x.StatusCode == GoodInterpolated);
s[0].SourceTimestampUtc.ShouldBe(At(5)); // stamped at the requested (sub-ms) time
s[0].SourceTimestampUtc.ShouldBe(At(5)); // stamped at the requested (sub-µs) time itself, never rounded (M3)
}
[Fact]
public async Task No_previous_sample_is_BadNoData()
{
var db = await ReadTestDb.CreateAsync(fx);
await db.SeedAAsync();
var s = (await db.Source().ReadAtTimeAsync("A", [At(0).AddTicks(-1)], Ct)).Samples.Single();
await ConformanceData.SeedAAsync(B);
var before = At(0).AddTicks(-B.ResolutionTicks);
var s = (await B.CreateDataSource().ReadAtTimeAsync("A", [before], Ct)).Samples.Single();
s.Value.ShouldBeNull();
s.StatusCode.ShouldBe(BadNoData);
s.SourceTimestampUtc.ShouldBe(At(0).AddTicks(-1));
s.SourceTimestampUtc.ShouldBe(before);
}
[Fact]
public async Task A_bad_next_neighbour_holds_the_previous_value_uncertain()
{
var db = await ReadTestDb.CreateAsync(fx);
await db.SeedAAsync();
var s = (await db.Source().ReadAtTimeAsync("A", [At(15)], Ct)).Samples.Single();
await ConformanceData.SeedAAsync(B);
var s = (await B.CreateDataSource().ReadAtTimeAsync("A", [At(15)], Ct)).Samples.Single();
s.Value.ShouldBe(20d); // NOT 25 (interpolating toward the Bad 30)
s.StatusCode.ShouldBe(UncertainInterpolated);
}
@@ -81,9 +76,8 @@ public sealed class AtTimeReadTests(SqlHistorianFixture fx)
[Fact]
public async Task A_bad_previous_neighbour_is_inside_an_outage_and_keeps_its_bad_code()
{
var db = await ReadTestDb.CreateAsync(fx);
await db.SeedAAsync();
var s = (await db.Source().ReadAtTimeAsync("A", [At(25)], Ct)).Samples.Single();
await ConformanceData.SeedAAsync(B);
var s = (await B.CreateDataSource().ReadAtTimeAsync("A", [At(25)], Ct)).Samples.Single();
s.Value.ShouldBeNull(); // NOT 35 (30→40) and NOT 30 (a Bad value)
s.StatusCode.ShouldBe(0x80000000u); // the previous sample's Bad code — never upgraded to Uncertain
s.SourceTimestampUtc.ShouldBe(At(25));
@@ -92,12 +86,11 @@ public sealed class AtTimeReadTests(SqlHistorianFixture fx)
[Fact]
public async Task Interpolation_never_bridges_an_outage_marker()
{
var db = await ReadTestDb.CreateAsync(fx);
await db.SeedAsync("M",
(At(0), HistorianValue.FromDouble(10), ReadTestDb.Good),
(At(5), HistorianValue.Empty, ReadTestDb.BadNoCommunication),
(At(10), HistorianValue.FromDouble(30), ReadTestDb.Good));
var s = (await db.Source().ReadAtTimeAsync("M", [At(2.5), At(7.5), At(5)], Ct)).Samples;
await B.SeedAsync("M",
(At(0), HistorianValue.FromDouble(10), Good),
(At(5), HistorianValue.Empty, BadNoCommunication),
(At(10), HistorianValue.FromDouble(30), Good));
var s = (await B.CreateDataSource().ReadAtTimeAsync("M", [At(2.5), At(7.5), At(5)], Ct)).Samples;
s[0].Value.ShouldBe(10d); // bridging 10→30 would give 15
s[0].StatusCode.ShouldBe(UncertainInterpolated);
s[1].Value.ShouldBeNull(); // bridging would give 25
@@ -109,9 +102,8 @@ public sealed class AtTimeReadTests(SqlHistorianFixture fx)
[Fact]
public async Task After_the_last_sample_the_value_is_stepped()
{
var db = await ReadTestDb.CreateAsync(fx);
await db.SeedAAsync();
var s = (await db.Source().ReadAtTimeAsync("A", [At(45)], Ct)).Samples.Single();
await ConformanceData.SeedAAsync(B);
var s = (await B.CreateDataSource().ReadAtTimeAsync("A", [At(45)], Ct)).Samples.Single();
s.Value.ShouldBe(50d);
s.StatusCode.ShouldBe(GoodInterpolated);
}
@@ -119,9 +111,8 @@ public sealed class AtTimeReadTests(SqlHistorianFixture fx)
[Fact]
public async Task A_time_after_now_stepped_from_the_last_sample_is_uncertain()
{
var db = await ReadTestDb.CreateAsync(fx);
await db.SeedAAsync(); // last sample +40 = 50, Good
var src = db.Source(new FixedClock(new DateTimeOffset(At(42))));
await ConformanceData.SeedAAsync(B); // last sample +40 = 50, Good
var src = B.CreateDataSource(new ConformanceClock(new DateTimeOffset(At(42))));
var s = (await src.ReadAtTimeAsync("A", [At(41), At(45)], Ct)).Samples;
s[0].Value.ShouldBe(50d);
s[0].StatusCode.ShouldBe(GoodInterpolated); // after the last sample but not after now: held Good
@@ -132,11 +123,10 @@ public sealed class AtTimeReadTests(SqlHistorianFixture fx)
[Fact]
public async Task An_uncertain_neighbour_makes_the_interpolation_uncertain()
{
var db = await ReadTestDb.CreateAsync(fx);
await db.SeedAsync("U",
(At(0), HistorianValue.FromDouble(10), ReadTestDb.Uncertain),
(At(10), HistorianValue.FromDouble(20), ReadTestDb.Good));
var s = (await db.Source().ReadAtTimeAsync("U", [At(5)], Ct)).Samples.Single();
await B.SeedAsync("U",
(At(0), HistorianValue.FromDouble(10), Uncertain),
(At(10), HistorianValue.FromDouble(20), Good));
var s = (await B.CreateDataSource().ReadAtTimeAsync("U", [At(5)], Ct)).Samples.Single();
s.Value.ShouldBe(15d);
s.StatusCode.ShouldBe(UncertainInterpolated);
}
@@ -144,11 +134,10 @@ public sealed class AtTimeReadTests(SqlHistorianFixture fx)
[Fact]
public async Task Int64_is_interpolated_and_rounded_keeping_its_type()
{
var db = await ReadTestDb.CreateAsync(fx);
await db.SeedAsync("I",
(At(0), HistorianValue.FromInt64(10), ReadTestDb.Good),
(At(10), HistorianValue.FromInt64(15), ReadTestDb.Good));
var s = (await db.Source().ReadAtTimeAsync("I", [At(5), At(2)], Ct)).Samples;
await B.SeedAsync("I",
(At(0), HistorianValue.FromInt64(10), Good),
(At(10), HistorianValue.FromInt64(15), Good));
var s = (await B.CreateDataSource().ReadAtTimeAsync("I", [At(5), At(2)], Ct)).Samples;
s[0].Value.ShouldBe(13L); // 12.5 ⇒ 13 (away from zero)
s[1].Value.ShouldBe(11L); // 11.0
s[0].StatusCode.ShouldBe(GoodInterpolated);
@@ -157,18 +146,17 @@ public sealed class AtTimeReadTests(SqlHistorianFixture fx)
[Fact]
public async Task Boolean_string_and_datetime_are_stepped_from_the_previous_sample()
{
var db = await ReadTestDb.CreateAsync(fx);
var d0 = new DateTime(2020, 1, 1, 0, 0, 0, DateTimeKind.Utc).AddTicks(3);
await db.SeedAsync("B",
(At(0), HistorianValue.FromBoolean(true), ReadTestDb.Good),
(At(10), HistorianValue.FromBoolean(false), ReadTestDb.Good));
await db.SeedAsync("S",
(At(0), HistorianValue.FromString("alpha"), ReadTestDb.Uncertain),
(At(10), HistorianValue.FromString("beta"), ReadTestDb.Good));
await db.SeedAsync("D",
(At(0), HistorianValue.FromDateTime(d0), ReadTestDb.Good),
(At(10), HistorianValue.FromDateTime(d0.AddDays(1)), ReadTestDb.Good));
var src = db.Source();
var d0 = new DateTime(2020, 1, 1, 0, 0, 0, DateTimeKind.Utc).AddTicks(3 * B.ResolutionTicks);
await B.SeedAsync("B",
(At(0), HistorianValue.FromBoolean(true), Good),
(At(10), HistorianValue.FromBoolean(false), Good));
await B.SeedAsync("S",
(At(0), HistorianValue.FromString("alpha"), Uncertain),
(At(10), HistorianValue.FromString("beta"), Good));
await B.SeedAsync("D",
(At(0), HistorianValue.FromDateTime(d0), Good),
(At(10), HistorianValue.FromDateTime(d0.AddDays(1)), Good));
var src = B.CreateDataSource();
var b = (await src.ReadAtTimeAsync("B", [At(9)], Ct)).Samples.Single();
b.Value.ShouldBe(true);
b.StatusCode.ShouldBe(GoodInterpolated);
@@ -181,9 +169,8 @@ public sealed class AtTimeReadTests(SqlHistorianFixture fx)
[Fact]
public async Task Result_matches_request_length_and_order_including_duplicates()
{
var db = await ReadTestDb.CreateAsync(fx);
await db.SeedAAsync();
var s = (await db.Source().ReadAtTimeAsync("A", [At(5), At(-1), At(5), At(10)], Ct)).Samples;
await ConformanceData.SeedAAsync(B);
var s = (await B.CreateDataSource().ReadAtTimeAsync("A", [At(5), At(-1), At(5), At(10)], Ct)).Samples;
s.Count.ShouldBe(4);
s.Select(x => x.Value).ShouldBe([15d, null, 15d, 20d]);
s.Select(x => x.SourceTimestampUtc).ShouldBe([At(5), At(-1), At(5), At(10)]);
@@ -192,8 +179,7 @@ public sealed class AtTimeReadTests(SqlHistorianFixture fx)
[Fact]
public async Task Empty_request_and_unknown_tag()
{
var db = await ReadTestDb.CreateAsync(fx);
var src = db.Source();
var src = B.CreateDataSource();
(await src.ReadAtTimeAsync("A", [], Ct)).Samples.ShouldBeEmpty();
var s = (await src.ReadAtTimeAsync("nope", [At(1), At(2)], Ct)).Samples;
s.Count.ShouldBe(2);
tests/Drivers/Historian.Conformance/ConformanceData.cs
+40
View File
@@ -0,0 +1,40 @@
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Conformance;
/// <summary>Shared fixture data of the conformance suite. <see cref="T0"/> is deliberately NOT millisecond- or microsecond-aligned.</summary>
public static class ConformanceData
{
/// <summary>Base time: 2026-10-09 08:00:00.1234567 UTC (sub-millisecond ticks on purpose).</summary>
public static readonly DateTime T0 = new DateTime(2026, 10, 9, 8, 0, 0, DateTimeKind.Utc).AddTicks(1_234_567);
/// <summary>Good.</summary>
public const uint Good = 0x00000000u;
/// <summary>Bad (generic).</summary>
public const uint Bad = 0x80000000u;
/// <summary>BadNoCommunication.</summary>
public const uint BadNoCommunication = 0x80310000u;
/// <summary>Uncertain (generic).</summary>
public const uint Uncertain = 0x40000000u;
/// <summary>T0 + <paramref name="seconds"/> seconds.</summary>
public static DateTime At(double seconds) => T0.AddTicks((long)(seconds * TimeSpan.TicksPerSecond));
/// <summary>The fixture tag A: +0=10, +10=20, +20=Bad(30), +30=40, +40=50 (Double).</summary>
public static Task SeedAAsync(IConformanceBackend b, string name = "A") => b.SeedAsync(name,
(At(0), HistorianValue.FromDouble(10), Good),
(At(10), HistorianValue.FromDouble(20), Good),
(At(20), HistorianValue.FromDouble(30), Bad),
(At(30), HistorianValue.FromDouble(40), Good),
(At(40), HistorianValue.FromDouble(50), Good));
}
/// <summary>A settable clock for a data source's "now".</summary>
public sealed class ConformanceClock(DateTimeOffset now) : TimeProvider
{
/// <summary>The current instant.</summary>
public DateTimeOffset Now { get; set; } = now;
/// <inheritdoc/>
public override DateTimeOffset GetUtcNow() => Now;
}
tests/Drivers/Historian.Conformance/IConformanceBackend.cs
+89
View File
@@ -0,0 +1,89 @@
using Xunit;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Conformance;
/// <summary>
/// One historian backend under test. The conformance suite talks to a backend only through this contract
/// (plus Core.Abstractions), so the same hand-computed expectations run against SQL Server and QuestDB.
/// Each instance owns an isolated store (a fresh schema / table set) so tests never see each other's data.
/// </summary>
public interface IConformanceBackend : IAsyncDisposable
{
/// <summary>Storage timestamp resolution in .NET ticks: 1 for SQL Server (100 ns), 10 for QuestDB (µs).</summary>
long ResolutionTicks { get; }
/// <summary>Truncates a timestamp to the backend's storage resolution (UTC kind). Every timestamp assertion goes through it.</summary>
/// <param name="t">The timestamp.</param>
/// <returns>The value the backend stores/serves for <paramref name="t"/>.</returns>
DateTime Round(DateTime t) => new(t.Ticks - t.Ticks % ResolutionTicks, DateTimeKind.Utc);
/// <summary>A fresh data source (fresh catalog cache) over this backend's store.</summary>
/// <param name="clock">Its "now"; null means the system clock.</param>
IHistorianDataSource CreateDataSource(TimeProvider? clock = null);
/// <summary>A fresh live-value writer over this backend's store.</summary>
/// <param name="utcNow">The clock used to stamp values that carry no timestamp; null means the system clock.</param>
IHistorianValueWriter CreateValueWriter(Func<DateTime>? utcNow = null);
/// <summary>A fresh history updater over this backend's store.</summary>
IHistorianDataUpdater CreateUpdater();
/// <summary>
/// Writes rows directly in the backend's schema (never through the writer or updater), status verbatim, creating the
/// tag if needed. The stored server timestamp of each row is its source time + 1 s, so tests can tell them apart.
/// </summary>
/// <param name="tag">The tag name.</param>
/// <param name="samples">The rows to store.</param>
Task SeedAsync(string tag, params (DateTime T, HistorianValue V, uint Status)[] samples);
/// <summary>
/// Live (non-deleted) rows for a tag, ascending by time, read directly from storage. The status is returned
/// VERBATIM as stored — there is no read-side clamp (an Empty row stored non-Bad reads back non-Bad here).
/// </summary>
/// <param name="tag">The tag name.</param>
Task<IReadOnlyList<(DateTime T, HistorianValue V, uint Status)>> ReadBackAsync(string tag);
/// <summary>Whether a tag with exactly this name exists in storage (no truncation of long names).</summary>
/// <param name="tag">The tag name.</param>
Task<bool> TagExistsAsync(string tag);
/// <summary>The data type the backend recorded for a tag, or null when the tag does not exist.</summary>
/// <param name="tag">The tag name.</param>
Task<DriverDataType?> TagDataTypeAsync(string tag);
}
/// <summary>
/// Base of every conformance class. A backend's test project subclasses each abstract suite class once,
/// supplying <see cref="CreateBackendAsync"/>, which must call its fixture's skip gate so the tests report Skipped
/// (not Failed) when the backend is unavailable.
/// </summary>
public abstract class ConformanceBase : IAsyncLifetime
{
/// <summary>Good.</summary>
protected const uint Good = ConformanceData.Good;
/// <summary>Bad (generic).</summary>
protected const uint Bad = ConformanceData.Bad;
/// <summary>BadNoCommunication.</summary>
protected const uint BadNoCommunication = ConformanceData.BadNoCommunication;
/// <summary>Uncertain (generic).</summary>
protected const uint Uncertain = ConformanceData.Uncertain;
/// <summary>The backend under test for the current test.</summary>
protected IConformanceBackend B { get; private set; } = null!;
/// <summary>Creates this test's isolated backend; calls the fixture's skip gate first.</summary>
protected abstract Task<IConformanceBackend> CreateBackendAsync();
/// <summary>T0 + <paramref name="s"/> seconds.</summary>
protected static DateTime At(double s) => ConformanceData.At(s);
/// <inheritdoc/>
public async ValueTask InitializeAsync() => B = await CreateBackendAsync();
/// <inheritdoc/>
public async ValueTask DisposeAsync()
{
if (B is not null) await B.DisposeAsync();
}
}
tests/Drivers/Historian.Conformance/LiveWriteConformance.cs
+97
View File
@@ -0,0 +1,97 @@
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Conformance;
/// <summary>Live value writer behaviour: round-trips, idempotent redelivery, first-wins duplicates, non-finite doubles.</summary>
public abstract class LiveWriteConformance : ConformanceBase
{
// Non-ms-aligned: sub-millisecond ticks must round-trip exactly (to the backend's resolution).
private static readonly DateTime T0 = new DateTime(2026, 10, 9, 1, 2, 3, DateTimeKind.Utc).AddTicks(1234567);
[Fact]
public async Task Every_kind_round_trips_with_exact_ticks_and_status()
{
var w = B.CreateValueWriter();
var r = B.ResolutionTicks;
var dt = new DateTime(2026, 1, 2, 3, 4, 5, DateTimeKind.Utc).AddTicks(77 * r);
HistorizationValue[] vals =
[
new(T0, HistorianValue.FromDouble(1.25), 0),
new(T0.AddTicks(1 * r), HistorianValue.FromInt64(long.MaxValue), 0x40000000u),
new(T0.AddTicks(2 * r), HistorianValue.FromBoolean(true), 0),
new(T0.AddTicks(3 * r), HistorianValue.FromString("héllo"), 0),
new(T0.AddTicks(4 * r), HistorianValue.FromDateTime(dt), 0),
new(T0.AddTicks(5 * r), HistorianValue.Empty, 0x80310000u),
];
(await w.WriteLiveValuesAsync("tag.a", vals, CancellationToken.None)).ShouldBeTrue();
var rows = await B.ReadBackAsync("tag.a");
rows.Count.ShouldBe(6);
for (var i = 0; i < 6; i++) rows[i].T.Ticks.ShouldBe(B.Round(vals[i].TimestampUtc!.Value).Ticks);
rows.Select(x => x.V).ToList().ShouldBe(vals.Select(v => v.Value).ToList());
rows[1].Status.ShouldBe(0x40000000u);
rows[5].Status.ShouldBe(0x80310000u);
}
[Fact]
public async Task Writing_the_same_batch_twice_is_idempotent()
{
var w = B.CreateValueWriter();
HistorizationValue[] vals = [new(T0, HistorianValue.FromDouble(1), 0), new(T0.AddSeconds(1), HistorianValue.FromDouble(2), 0)];
(await w.WriteLiveValuesAsync("tag.b", vals, CancellationToken.None)).ShouldBeTrue();
(await w.WriteLiveValuesAsync("tag.b", vals, CancellationToken.None)).ShouldBeTrue();
// Overlapping batch: only the new row lands, the stored first value is untouched.
(await w.WriteLiveValuesAsync("tag.b", [new(T0, HistorianValue.FromDouble(99), 0), new(T0.AddSeconds(2), HistorianValue.FromDouble(3), 0)], CancellationToken.None)).ShouldBeTrue();
var rows = await B.ReadBackAsync("tag.b");
rows.Count.ShouldBe(3);
rows[0].V.Number.ShouldBe(1);
}
[Fact]
public async Task Within_batch_duplicate_timestamps_do_not_fail_and_first_wins()
{
var w = B.CreateValueWriter();
(await w.WriteLiveValuesAsync("tag.c", [new(T0, HistorianValue.FromDouble(1), 0), new(T0, HistorianValue.FromDouble(2), 0)], CancellationToken.None)).ShouldBeTrue();
var rows = await B.ReadBackAsync("tag.c");
rows.Count.ShouldBe(1);
rows[0].V.Number.ShouldBe(1);
}
[Fact]
public async Task Non_finite_doubles_store_as_Empty_forced_Bad_and_do_not_fail_the_write()
{
var w = B.CreateValueWriter();
var r = B.ResolutionTicks;
(await w.WriteLiveValuesAsync("tag.d", [
new(T0, HistorianValue.FromDouble(double.NaN), 0x40000000u),
new(T0.AddTicks(1 * r), HistorianValue.FromDouble(double.PositiveInfinity), 0),
new(T0.AddTicks(2 * r), HistorianValue.FromDouble(7), 0)], CancellationToken.None)).ShouldBeTrue();
var rows = await B.ReadBackAsync("tag.d");
rows.Count.ShouldBe(3);
rows[0].V.Kind.ShouldBe(HistorianValueKind.Empty);
rows[0].Status.ShouldBe(HistorianStatusCodes.BadOutOfRange); // Uncertain NaN ⇒ Bad (I-1)
rows[1].Status.ShouldBe(HistorianStatusCodes.BadOutOfRange); // Good +Inf ⇒ Bad (I-1)
rows[2].V.Number.ShouldBe(7);
}
[Fact]
public async Task Empty_batch_succeeds_without_a_call_and_null_timestamp_is_stamped()
{
var w = B.CreateValueWriter();
(await w.WriteLiveValuesAsync("tag.e", [], CancellationToken.None)).ShouldBeTrue();
(await w.WriteLiveValuesAsync("tag.e", [new(null, HistorianValue.FromInt64(5), 0)], CancellationToken.None)).ShouldBeTrue();
var rows = await B.ReadBackAsync("tag.e");
rows.Count.ShouldBe(1);
(DateTime.UtcNow - rows[0].T).ShouldBeLessThan(TimeSpan.FromMinutes(1));
}
[Fact]
public async Task Tag_is_created_with_the_first_typed_values_data_type()
{
var w = B.CreateValueWriter();
await w.WriteLiveValuesAsync("tag.f", [new(T0, HistorianValue.Empty, 0x80310000u), new(T0.AddTicks(B.ResolutionTicks), HistorianValue.FromString("x"), 0)], CancellationToken.None);
(await B.TagDataTypeAsync("tag.f")).ShouldBe(DriverDataType.String);
}
}
tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/ProcessedReadTests.cs → tests/Drivers/Historian.Conformance/ProcessedConformance.cs
+80 -110
View File
@@ -2,20 +2,18 @@ using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Conformance;
/// <summary>
/// Push-down aggregates over tag A (+0=10, +10=20, +20=Bad(30), +30=40, +40=50), 20 s intervals from the
/// sub-millisecond <see cref="ReadTestDb.T0"/>, window [t0, t0+80s) ⇒ four buckets:
/// sub-millisecond <see cref="ConformanceData.T0"/>, window [t0, t0+80s) ⇒ four buckets:
/// B0 [0,20) = {10, 20}; B1 [20,40) = {Bad 30, 40}; B2 [40,60) = {50}; B3 [60,80) = {} (empty).
/// Every expected value below is hand-computed.
/// </summary>
[Collection(SqlHistorianCollection.Name)]
public sealed class ProcessedReadTests(SqlHistorianFixture fx)
public abstract class ProcessedConformance : ConformanceBase
{
private static readonly CancellationToken Ct = CancellationToken.None;
private static readonly TimeSpan Twenty = TimeSpan.FromSeconds(20);
private static DateTime At(double s) => ReadTestDb.At(s);
// Calculated = InfoType DataValue 0x400 | Calculated 0x1, OR-ed onto the code.
private const uint GoodCalc = 0x00000401u;
@@ -23,8 +21,8 @@ public sealed class ProcessedReadTests(SqlHistorianFixture fx)
private const uint NoDataCalc = 0x809B0401u; // BadNoData | 0x401
private const uint InvalidInputsCalc = 0x80D60401u; // BadAggregateInvalidInputs | 0x401
private static async Task<HistoryReadResult> ReadA(ReadTestDb db, HistoryAggregateType agg) =>
await db.Source().ReadProcessedAsync("A", At(0), At(80), Twenty, agg, Ct);
private async Task<HistoryReadResult> ReadA(HistoryAggregateType agg) =>
await B.CreateDataSource().ReadProcessedAsync("A", At(0), At(80), Twenty, agg, Ct);
[Fact]
public void The_status_literals_carry_the_SDK_Calculated_aggregate_bits()
@@ -48,23 +46,21 @@ public sealed class ProcessedReadTests(SqlHistorianFixture fx)
[InlineData(HistoryAggregateType.Range, 10d, 0d, 0d)]
public async Task Numeric_aggregates_over_the_fixture(HistoryAggregateType agg, double b0, double b1, double b2)
{
var db = await ReadTestDb.CreateAsync(fx);
await db.SeedAAsync();
var s = (await ReadA(db, agg)).Samples;
await ConformanceData.SeedAAsync(B);
var s = (await ReadA(agg)).Samples;
s.Count.ShouldBe(4);
s.Select(x => x.Value).ShouldBe([b0, b1, b2, null]);
// B1 holds a Bad sample alongside a usable one ⇒ Uncertain; the empty B3 ⇒ BadNoData.
s.Select(x => x.StatusCode).ShouldBe([GoodCalc, UncertainCalc, GoodCalc, NoDataCalc]);
s.Select(x => x.SourceTimestampUtc).ShouldBe([At(0), At(20), At(40), At(60)]); // bucket starts
s.Select(x => x.SourceTimestampUtc).ShouldBe([B.Round(At(0)), B.Round(At(20)), B.Round(At(40)), B.Round(At(60))]); // bucket starts
}
[Fact]
public async Task Count_counts_usable_samples_as_int()
{
var db = await ReadTestDb.CreateAsync(fx);
await db.SeedAAsync();
var s = (await ReadA(db, HistoryAggregateType.Count)).Samples;
await ConformanceData.SeedAAsync(B);
var s = (await ReadA(HistoryAggregateType.Count)).Samples;
s.Select(x => x.Value).ShouldBe([(object?)2, 1, 1, null]);
s.Select(x => x.StatusCode).ShouldBe([GoodCalc, UncertainCalc, GoodCalc, NoDataCalc]);
}
@@ -74,9 +70,8 @@ public sealed class ProcessedReadTests(SqlHistorianFixture fx)
[InlineData(HistoryAggregateType.PercentBad, 0d, 50d, 0d)]
public async Task Percent_aggregates_count_every_sample(HistoryAggregateType agg, double b0, double b1, double b2)
{
var db = await ReadTestDb.CreateAsync(fx);
await db.SeedAAsync();
var s = (await ReadA(db, agg)).Samples;
await ConformanceData.SeedAAsync(B);
var s = (await ReadA(agg)).Samples;
s.Select(x => x.Value).ShouldBe([b0, b1, b2, null]);
// A percentage is a statement about quality, so it is not itself degraded by the Bad input.
s.Select(x => x.StatusCode).ShouldBe([GoodCalc, GoodCalc, GoodCalc, NoDataCalc]);
@@ -85,34 +80,32 @@ public sealed class ProcessedReadTests(SqlHistorianFixture fx)
[Fact]
public async Task Actual_time_aggregates_stamp_the_sample_time()
{
var db = await ReadTestDb.CreateAsync(fx);
await db.SeedAAsync();
var min = (await ReadA(db, HistoryAggregateType.MinimumActualTime)).Samples;
await ConformanceData.SeedAAsync(B);
var min = (await ReadA(HistoryAggregateType.MinimumActualTime)).Samples;
min.Select(x => x.Value).ShouldBe([10d, 40d, 50d, null]);
min.Select(x => x.SourceTimestampUtc).ShouldBe([At(0), At(30), At(40), At(60)]);
min.Select(x => x.SourceTimestampUtc).ShouldBe([B.Round(At(0)), B.Round(At(30)), B.Round(At(40)), B.Round(At(60))]);
min.Select(x => x.StatusCode).ShouldBe([GoodCalc, UncertainCalc, GoodCalc, NoDataCalc]);
var max = (await ReadA(db, HistoryAggregateType.MaximumActualTime)).Samples;
var max = (await ReadA(HistoryAggregateType.MaximumActualTime)).Samples;
max.Select(x => x.Value).ShouldBe([20d, 40d, 50d, null]);
max.Select(x => x.SourceTimestampUtc).ShouldBe([At(10), At(30), At(40), At(60)]);
max.Select(x => x.SourceTimestampUtc).ShouldBe([B.Round(At(10)), B.Round(At(30)), B.Round(At(40)), B.Round(At(60))]);
}
[Fact]
public async Task Actual_time_ties_resolve_to_the_earliest_sample()
{
var db = await ReadTestDb.CreateAsync(fx);
await db.SeedAsync("T",
(At(0), HistorianValue.FromDouble(5), ReadTestDb.Good),
(At(2), HistorianValue.FromDouble(3), ReadTestDb.Good),
(At(4), HistorianValue.FromDouble(3), ReadTestDb.Good),
(At(6), HistorianValue.FromDouble(7), ReadTestDb.Good),
(At(8), HistorianValue.FromDouble(7), ReadTestDb.Good));
var src = db.Source();
await B.SeedAsync("T",
(At(0), HistorianValue.FromDouble(5), Good),
(At(2), HistorianValue.FromDouble(3), Good),
(At(4), HistorianValue.FromDouble(3), Good),
(At(6), HistorianValue.FromDouble(7), Good),
(At(8), HistorianValue.FromDouble(7), Good));
var src = B.CreateDataSource();
var min = (await src.ReadProcessedAsync("T", At(0), At(10), TimeSpan.Zero, HistoryAggregateType.MinimumActualTime, Ct)).Samples.Single();
min.Value.ShouldBe(3d);
min.SourceTimestampUtc.ShouldBe(At(2));
min.SourceTimestampUtc.ShouldBe(B.Round(At(2)));
var max = (await src.ReadProcessedAsync("T", At(0), At(10), TimeSpan.Zero, HistoryAggregateType.MaximumActualTime, Ct)).Samples.Single();
max.Value.ShouldBe(7d);
max.SourceTimestampUtc.ShouldBe(At(6));
max.SourceTimestampUtc.ShouldBe(B.Round(At(6)));
}
[Fact]
@@ -120,16 +113,15 @@ public sealed class ProcessedReadTests(SqlHistorianFixture fx)
{
// T0 is not millisecond-aligned: a DATEDIFF(millisecond) bucket formula would push the sample one tick
// before the +20 s edge into bucket 1 (it counts millisecond boundaries crossed, not elapsed time).
var db = await ReadTestDb.CreateAsync(fx);
var edge = At(20);
await db.SeedAsync("E",
(At(0).AddTicks(-1), HistorianValue.FromDouble(1000), ReadTestDb.Good), // before start: excluded
(At(0), HistorianValue.FromDouble(1), ReadTestDb.Good), // == start: B0
(edge.AddTicks(-1), HistorianValue.FromDouble(2), ReadTestDb.Good), // last tick of B0
(edge, HistorianValue.FromDouble(4), ReadTestDb.Good), // first tick of B1
(At(40).AddTicks(-1), HistorianValue.FromDouble(8), ReadTestDb.Good), // last tick of B1
(At(40), HistorianValue.FromDouble(2000), ReadTestDb.Good)); // == end: excluded
var s = (await db.Source().ReadProcessedAsync("E", At(0), At(40), Twenty, HistoryAggregateType.Total, Ct)).Samples;
await B.SeedAsync("E",
(At(0).AddTicks(-B.ResolutionTicks), HistorianValue.FromDouble(1000), Good), // before start: excluded
(At(0), HistorianValue.FromDouble(1), Good), // == start: B0
(edge.AddTicks(-B.ResolutionTicks), HistorianValue.FromDouble(2), Good), // last tick of B0
(edge, HistorianValue.FromDouble(4), Good), // first tick of B1
(At(40).AddTicks(-B.ResolutionTicks), HistorianValue.FromDouble(8), Good), // last tick of B1
(At(40), HistorianValue.FromDouble(2000), Good)); // == end: excluded
var s = (await B.CreateDataSource().ReadProcessedAsync("E", At(0), At(40), Twenty, HistoryAggregateType.Total, Ct)).Samples;
s.Select(x => x.Value).ShouldBe([3d, 12d]);
s.ShouldAllBe(x => x.StatusCode == GoodCalc);
}
@@ -137,23 +129,21 @@ public sealed class ProcessedReadTests(SqlHistorianFixture fx)
[Fact]
public async Task A_partial_last_interval_is_truncated_at_end()
{
var db = await ReadTestDb.CreateAsync(fx);
await db.SeedAAsync();
var s = (await db.Source().ReadProcessedAsync("A", At(0), At(50), Twenty, HistoryAggregateType.Average, Ct)).Samples;
s.Select(x => x.SourceTimestampUtc).ShouldBe([At(0), At(20), At(40)]);
await ConformanceData.SeedAAsync(B);
var s = (await B.CreateDataSource().ReadProcessedAsync("A", At(0), At(50), Twenty, HistoryAggregateType.Average, Ct)).Samples;
s.Select(x => x.SourceTimestampUtc).ShouldBe([B.Round(At(0)), B.Round(At(20)), B.Round(At(40))]);
s.Select(x => x.Value).ShouldBe([15d, 40d, 50d]);
}
[Fact]
public async Task A_non_positive_interval_is_one_interval_over_the_window()
{
var db = await ReadTestDb.CreateAsync(fx);
await db.SeedAAsync();
var src = db.Source();
await ConformanceData.SeedAAsync(B);
var src = B.CreateDataSource();
var avg = (await src.ReadProcessedAsync("A", At(0), At(50), TimeSpan.Zero, HistoryAggregateType.Average, Ct)).Samples.Single();
avg.Value.ShouldBe(30d); // usable {10, 20, 40, 50}
avg.StatusCode.ShouldBe(UncertainCalc);
avg.SourceTimestampUtc.ShouldBe(At(0));
avg.SourceTimestampUtc.ShouldBe(B.Round(At(0)));
(await src.ReadProcessedAsync("A", At(0), At(50), TimeSpan.FromSeconds(-1), HistoryAggregateType.Count, Ct))
.Samples.Single().Value.ShouldBe(4);
}
@@ -163,11 +153,10 @@ public sealed class ProcessedReadTests(SqlHistorianFixture fx)
{
// 1601-01-01 is the OPC UA minimum DateTime. A nanosecond DATEDIFF_BIG overflows past ≈292 years (SQL 535),
// and a ns-scaled interval would wrap a long; both must be gone.
var db = await ReadTestDb.CreateAsync(fx);
await db.SeedAAsync();
await ConformanceData.SeedAAsync(B);
var start = new DateTime(1601, 1, 1, 0, 0, 0, DateTimeKind.Utc);
var end = new DateTime(9999, 12, 31, 0, 0, 0, DateTimeKind.Utc);
var src = db.Source();
var src = B.CreateDataSource();
var avg = (await src.ReadProcessedAsync("A", start, end, TimeSpan.Zero, HistoryAggregateType.Average, Ct)).Samples.Single();
avg.Value.ShouldBe(30d); // usable {10, 20, 40, 50}
avg.StatusCode.ShouldBe(UncertainCalc);
@@ -189,11 +178,10 @@ public sealed class ProcessedReadTests(SqlHistorianFixture fx)
// Hand-computed (calendar arithmetic, independent of SQL): 1601-01-01 → 2027-01-01 is 155 593 days =
// 22 227.57 weeks ⇒ 22 228 intervals (under the 100k cap). 2026-10-09 08:00 is 155 509 days in ⇒ week
// index 22 215, which starts Monday 2026-10-05 00:00; every fixture sample (t0 … t0+40 s) is in it.
var db = await ReadTestDb.CreateAsync(fx);
await db.SeedAAsync();
await ConformanceData.SeedAAsync(B);
var start = new DateTime(1601, 1, 1, 0, 0, 0, DateTimeKind.Utc);
var end = new DateTime(2027, 1, 1, 0, 0, 0, DateTimeKind.Utc);
var s = (await db.Source().ReadProcessedAsync("A", start, end, TimeSpan.FromDays(7), HistoryAggregateType.Average, Ct)).Samples;
var s = (await B.CreateDataSource().ReadProcessedAsync("A", start, end, TimeSpan.FromDays(7), HistoryAggregateType.Average, Ct)).Samples;
s.Count.ShouldBe(22_228);
var hit = s[22_215];
@@ -207,31 +195,28 @@ public sealed class ProcessedReadTests(SqlHistorianFixture fx)
[Fact]
public async Task A_reverse_window_is_computed_forward_and_returned_descending()
{
var db = await ReadTestDb.CreateAsync(fx);
await db.SeedAAsync();
var s = (await db.Source().ReadProcessedAsync("A", At(80), At(0), Twenty, HistoryAggregateType.Average, Ct)).Samples;
s.Select(x => x.SourceTimestampUtc).ShouldBe([At(60), At(40), At(20), At(0)]);
await ConformanceData.SeedAAsync(B);
var s = (await B.CreateDataSource().ReadProcessedAsync("A", At(80), At(0), Twenty, HistoryAggregateType.Average, Ct)).Samples;
s.Select(x => x.SourceTimestampUtc).ShouldBe([B.Round(At(60)), B.Round(At(40)), B.Round(At(20)), B.Round(At(0))]);
s.Select(x => x.Value).ShouldBe([null, 50d, 40d, 15d]);
}
[Fact]
public async Task Equal_start_and_end_yields_no_intervals()
{
var db = await ReadTestDb.CreateAsync(fx);
await db.SeedAAsync();
(await db.Source().ReadProcessedAsync("A", At(10), At(10), Twenty, HistoryAggregateType.Average, Ct))
await ConformanceData.SeedAAsync(B);
(await B.CreateDataSource().ReadProcessedAsync("A", At(10), At(10), Twenty, HistoryAggregateType.Average, Ct))
.Samples.ShouldBeEmpty();
}
[Fact]
public async Task Outage_markers_are_bad_samples_in_aggregates()
{
var db = await ReadTestDb.CreateAsync(fx);
await db.SeedAsync("M",
(At(0), HistorianValue.FromDouble(10), ReadTestDb.Good),
(At(5), HistorianValue.Empty, ReadTestDb.BadNoCommunication),
(At(10), HistorianValue.FromDouble(30), ReadTestDb.Good));
var src = db.Source();
await B.SeedAsync("M",
(At(0), HistorianValue.FromDouble(10), Good),
(At(5), HistorianValue.Empty, BadNoCommunication),
(At(10), HistorianValue.FromDouble(30), Good));
var src = B.CreateDataSource();
async Task<DataValueSnapshot> One(HistoryAggregateType a) =>
(await src.ReadProcessedAsync("M", At(0), At(20), Twenty, a, Ct)).Samples.Single();
@@ -246,11 +231,10 @@ public sealed class ProcessedReadTests(SqlHistorianFixture fx)
[Fact]
public async Task An_interval_of_only_bad_samples_is_BadNoData_except_the_percentages()
{
var db = await ReadTestDb.CreateAsync(fx);
await db.SeedAsync("Z",
(At(1), HistorianValue.Empty, ReadTestDb.BadNoCommunication),
(At(2), HistorianValue.FromDouble(5), ReadTestDb.Bad));
var src = db.Source();
await B.SeedAsync("Z",
(At(1), HistorianValue.Empty, BadNoCommunication),
(At(2), HistorianValue.FromDouble(5), Bad));
var src = B.CreateDataSource();
async Task<DataValueSnapshot> One(HistoryAggregateType a) =>
(await src.ReadProcessedAsync("Z", At(0), At(20), Twenty, a, Ct)).Samples.Single();
@@ -264,11 +248,10 @@ public sealed class ProcessedReadTests(SqlHistorianFixture fx)
[Fact]
public async Task Uncertain_samples_are_used_but_make_the_interval_uncertain()
{
var db = await ReadTestDb.CreateAsync(fx);
await db.SeedAsync("U",
(At(1), HistorianValue.FromDouble(4), ReadTestDb.Uncertain),
(At(2), HistorianValue.FromDouble(6), ReadTestDb.Good));
var s = (await db.Source().ReadProcessedAsync("U", At(0), At(20), Twenty, HistoryAggregateType.Average, Ct)).Samples.Single();
await B.SeedAsync("U",
(At(1), HistorianValue.FromDouble(4), Uncertain),
(At(2), HistorianValue.FromDouble(6), Good));
var s = (await B.CreateDataSource().ReadProcessedAsync("U", At(0), At(20), Twenty, HistoryAggregateType.Average, Ct)).Samples.Single();
s.Value.ShouldBe(5d);
s.StatusCode.ShouldBe(UncertainCalc);
}
@@ -276,11 +259,10 @@ public sealed class ProcessedReadTests(SqlHistorianFixture fx)
[Fact]
public async Task String_tags_serve_count_start_end_and_refuse_numeric_aggregates()
{
var db = await ReadTestDb.CreateAsync(fx);
await db.SeedAsync("S",
(At(0), HistorianValue.FromString("a"), ReadTestDb.Good),
(At(10), HistorianValue.FromString("b"), ReadTestDb.Good));
var src = db.Source();
await B.SeedAsync("S",
(At(0), HistorianValue.FromString("a"), Good),
(At(10), HistorianValue.FromString("b"), Good));
var src = B.CreateDataSource();
async Task<DataValueSnapshot> One(HistoryAggregateType a) =>
(await src.ReadProcessedAsync("S", At(0), At(20), Twenty, a, Ct)).Samples.Single();
@@ -304,22 +286,20 @@ public sealed class ProcessedReadTests(SqlHistorianFixture fx)
[Fact]
public async Task DateTime_tags_refuse_numeric_aggregates()
{
var db = await ReadTestDb.CreateAsync(fx);
await db.SeedAsync("D", (At(0), HistorianValue.FromDateTime(At(-100)), ReadTestDb.Good));
(await db.Source().ReadProcessedAsync("D", At(0), At(20), Twenty, HistoryAggregateType.Average, Ct))
await B.SeedAsync("D", (At(0), HistorianValue.FromDateTime(At(-100)), Good));
(await B.CreateDataSource().ReadProcessedAsync("D", At(0), At(20), Twenty, HistoryAggregateType.Average, Ct))
.Samples.Single().StatusCode.ShouldBe(InvalidInputsCalc);
}
[Fact]
public async Task Boolean_tags_aggregate_as_zero_or_one()
{
var db = await ReadTestDb.CreateAsync(fx);
await db.SeedAsync("B",
(At(0), HistorianValue.FromBoolean(true), ReadTestDb.Good),
(At(5), HistorianValue.FromBoolean(false), ReadTestDb.Good),
(At(10), HistorianValue.FromBoolean(true), ReadTestDb.Good),
(At(15), HistorianValue.FromBoolean(true), ReadTestDb.Good));
var src = db.Source();
await B.SeedAsync("B",
(At(0), HistorianValue.FromBoolean(true), Good),
(At(5), HistorianValue.FromBoolean(false), Good),
(At(10), HistorianValue.FromBoolean(true), Good),
(At(15), HistorianValue.FromBoolean(true), Good));
var src = B.CreateDataSource();
async Task<DataValueSnapshot> One(HistoryAggregateType a) =>
(await src.ReadProcessedAsync("B", At(0), At(20), Twenty, a, Ct)).Samples.Single();
(await One(HistoryAggregateType.Average)).Value.ShouldBe(0.75d);
@@ -331,8 +311,7 @@ public sealed class ProcessedReadTests(SqlHistorianFixture fx)
[Fact]
public async Task Unknown_tag_yields_every_interval_as_BadNoData()
{
var db = await ReadTestDb.CreateAsync(fx);
var s = (await db.Source().ReadProcessedAsync("nope", At(0), At(60), Twenty, HistoryAggregateType.Average, Ct)).Samples;
var s = (await B.CreateDataSource().ReadProcessedAsync("nope", At(0), At(60), Twenty, HistoryAggregateType.Average, Ct)).Samples;
s.Count.ShouldBe(3);
s.ShouldAllBe(x => x.StatusCode == NoDataCalc && x.Value == null);
}
@@ -352,31 +331,22 @@ public sealed class ProcessedReadTests(SqlHistorianFixture fx)
[InlineData(HistoryAggregateType.VariancePopulation)]
public async Task Unsupported_aggregates_throw_not_supported_and_are_not_health_failures(HistoryAggregateType agg)
{
var db = await ReadTestDb.CreateAsync(fx);
var src = db.Source();
var src = B.CreateDataSource();
var ex = await Should.ThrowAsync<HistoryAggregateNotSupportedException>(
() => src.ReadProcessedAsync("A", At(0), At(20), Twenty, agg, Ct));
ex.Aggregate.ShouldBe(agg);
src.GetHealthSnapshot().TotalQueries.ShouldBe(0);
}
[Fact]
public void Every_enum_member_is_either_served_or_refused()
{
// Thirteen served; anything appended to the enum later falls to the refusal (total default).
Enum.GetValues<HistoryAggregateType>().Count(a => SqlAggregateQueries.FamilyOf(a) is not null).ShouldBe(13);
}
[Fact]
public async Task The_bucket_count_is_capped_at_100000()
{
var db = await ReadTestDb.CreateAsync(fx);
var src = db.Source();
var oneTick = TimeSpan.FromTicks(1);
(await src.ReadProcessedAsync("nope", At(0), At(0).AddTicks(100_000), oneTick, HistoryAggregateType.Count, Ct))
var src = B.CreateDataSource();
var oneTick = TimeSpan.FromTicks(B.ResolutionTicks);
(await src.ReadProcessedAsync("nope", At(0), At(0).AddTicks(100_000 * B.ResolutionTicks), oneTick, HistoryAggregateType.Count, Ct))
.Samples.Count.ShouldBe(100_000);
await Should.ThrowAsync<ArgumentOutOfRangeException>(
() => src.ReadProcessedAsync("nope", At(0), At(0).AddTicks(100_001), oneTick, HistoryAggregateType.Count, Ct));
() => src.ReadProcessedAsync("nope", At(0), At(0).AddTicks(100_001 * B.ResolutionTicks), oneTick, HistoryAggregateType.Count, Ct));
var h = src.GetHealthSnapshot();
h.TotalFailures.ShouldBe(0); // a client-request error, not a backend fault
}
tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/QualityContractTests.cs → tests/Drivers/Historian.Conformance/QualityContractConformance.cs
+25 -35
View File
@@ -1,52 +1,43 @@
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Conformance;
/// <summary>
/// Final-review I-1: a stored sample is never served with a better quality than "no value". An Empty (null-value)
/// sample is always Bad — forced Bad by the value writer, refused non-Bad by the updater, and clamped to
/// <c>BadNoData</c> on read for rows written before either rule existed.
/// </summary>
[Collection(SqlHistorianCollection.Name)]
public sealed class QualityContractTests(SqlHistorianFixture fx)
public abstract class QualityContractConformance : ConformanceBase
{
private static readonly CancellationToken Ct = CancellationToken.None;
private static DateTime At(double s) => ReadTestDb.At(s);
private const uint BadOutOfRange = 0x803C0000u;
private const uint BadNoData = 0x809B0000u;
private const uint BadInvalidArgument = 0x80AB0000u;
private const uint BadNoCommunication = 0x80310000u;
private static bool IsBad(uint status) => (status & 0x80000000u) != 0;
private static SqlServerHistorianValueWriter Writer(ReadTestDb db) =>
new(db.Conn, db.Boot, new SqlServerTagCatalog(db.Conn, db.Boot));
private static SqlServerHistorianDataUpdater Updater(ReadTestDb db) =>
new(db.Conn, new SqlServerTagCatalog(db.Conn, db.Boot));
[Fact]
public async Task A_good_NaN_written_through_the_real_writer_reads_back_Bad_on_Raw_and_AtTime()
{
var db = await ReadTestDb.CreateAsync(fx);
(await Writer(db).WriteLiveValuesAsync("nan", [
(await B.CreateValueWriter().WriteLiveValuesAsync("nan", [
new(At(0), HistorianValue.FromDouble(10), 0u),
new(At(10), HistorianValue.FromDouble(double.NaN), 0u),
new(At(20), HistorianValue.FromDouble(double.PositiveInfinity), 0x40000000u),
new(At(30), HistorianValue.FromDouble(double.NegativeInfinity), BadNoCommunication),
], Ct)).ShouldBeTrue();
var raw = (await db.Source().ReadRawAsync("nan", At(0), At(100), 0, Ct)).Samples;
var raw = (await B.CreateDataSource().ReadRawAsync("nan", At(0), At(100), 0, Ct)).Samples;
raw.Count.ShouldBe(4);
raw[1].Value.ShouldBeNull();
raw[1].StatusCode.ShouldBe(BadOutOfRange); // Good NaN ⇒ stored Bad
raw[2].StatusCode.ShouldBe(BadOutOfRange); // Uncertain +Inf ⇒ stored Bad
raw[3].StatusCode.ShouldBe(BadNoCommunication); // an existing Bad code is kept
var at = (await db.Source().ReadAtTimeAsync("nan", [At(10), At(15)], Ct)).Samples;
var at = (await B.CreateDataSource().ReadAtTimeAsync("nan", [At(10), At(15)], Ct)).Samples;
at.ShouldAllBe(s => s.Value == null);
at.ShouldAllBe(s => IsBad(s.StatusCode));
at[1].StatusCode.ShouldBe(BadOutOfRange); // inside the "outage" the NaN opened
@@ -55,50 +46,49 @@ public sealed class QualityContractTests(SqlHistorianFixture fx)
[Fact]
public async Task A_non_bad_Empty_written_through_the_real_writer_is_stored_BadNoData()
{
var db = await ReadTestDb.CreateAsync(fx);
(await Writer(db).WriteLiveValuesAsync("empty", [
(await B.CreateValueWriter().WriteLiveValuesAsync("empty", [
new(At(0), HistorianValue.Empty, 0u),
new(At(1), HistorianValue.Empty, 0x40000000u),
new(At(2), HistorianValue.Empty, BadNoCommunication),
], Ct)).ShouldBeTrue();
// Read the stored column directly so the read-side clamp cannot mask a writer that forgot to force Bad.
(await db.ScalarAsync($"SELECT COUNT(*) FROM {db.Conn.Q("Sample")} WHERE StatusCode >= 0")).ShouldBe(0);
var raw = (await db.Source().ReadRawAsync("empty", At(0), At(10), 0, Ct)).Samples;
// Read the stored status directly so the read-side clamp cannot mask a writer that forgot to force Bad.
var stored = await B.ReadBackAsync("empty");
stored.Count.ShouldBe(3);
stored.ShouldAllBe(r => IsBad(r.Status));
var raw = (await B.CreateDataSource().ReadRawAsync("empty", At(0), At(10), 0, Ct)).Samples;
raw.Select(s => s.StatusCode).ShouldBe([BadNoData, BadNoData, BadNoCommunication]);
}
[Fact]
public async Task A_legacy_good_Empty_row_is_clamped_to_BadNoData_on_Raw_and_AtTime()
{
var db = await ReadTestDb.CreateAsync(fx);
// Seeded with plain INSERTs, bypassing the writer: what a pre-fix build could have stored.
await db.SeedAsync("legacy",
(At(0), HistorianValue.FromDouble(10), ReadTestDb.Good),
(At(10), HistorianValue.Empty, ReadTestDb.Good),
(At(20), HistorianValue.Empty, ReadTestDb.Uncertain),
(At(30), HistorianValue.FromDouble(40), ReadTestDb.Good));
// Seeded directly, bypassing the writer: what a pre-fix build could have stored.
await B.SeedAsync("legacy",
(At(0), HistorianValue.FromDouble(10), Good),
(At(10), HistorianValue.Empty, Good),
(At(20), HistorianValue.Empty, Uncertain),
(At(30), HistorianValue.FromDouble(40), Good));
var raw = (await db.Source().ReadRawAsync("legacy", At(0), At(100), 0, Ct)).Samples;
raw.Select(s => s.StatusCode).ShouldBe([ReadTestDb.Good, BadNoData, BadNoData, ReadTestDb.Good]);
var raw = (await B.CreateDataSource().ReadRawAsync("legacy", At(0), At(100), 0, Ct)).Samples;
raw.Select(s => s.StatusCode).ShouldBe([Good, BadNoData, BadNoData, Good]);
var at = (await db.Source().ReadAtTimeAsync("legacy", [At(10), At(15), At(25)], Ct)).Samples;
var at = (await B.CreateDataSource().ReadAtTimeAsync("legacy", [At(10), At(15), At(25)], Ct)).Samples;
at.ShouldAllBe(s => s.Value == null && s.StatusCode == BadNoData);
}
[Fact]
public async Task The_updater_refuses_a_non_bad_Empty_and_accepts_a_bad_one()
{
var db = await ReadTestDb.CreateAsync(fx);
var r = await Updater(db).UpdateDataAsync("u", HistoryUpdateMode.Insert, [
var r = await B.CreateUpdater().UpdateDataAsync("u", HistoryUpdateMode.Insert, [
new(At(0), HistorianValue.Empty, 0u),
new(At(1), HistorianValue.Empty, 0x40000000u),
new(At(2), HistorianValue.Empty, BadNoCommunication),
new(At(3), HistorianValue.FromDouble(1), 0u),
], Ct);
r.ShouldBe([BadInvalidArgument, BadInvalidArgument, SqlHistorianStatusCodes.GoodEntryInserted,
SqlHistorianStatusCodes.GoodEntryInserted]);
(await db.Source().ReadRawAsync("u", At(0), At(10), 0, Ct)).Samples
r.ShouldBe([BadInvalidArgument, BadInvalidArgument, HistorianStatusCodes.GoodEntryInserted,
HistorianStatusCodes.GoodEntryInserted]);
(await B.CreateDataSource().ReadRawAsync("u", At(0), At(10), 0, Ct)).Samples
.Select(s => s.StatusCode).ShouldBe([BadNoCommunication, 0u]);
}
}
tests/Drivers/Historian.Conformance/RawReadConformance.cs
+134
View File
@@ -0,0 +1,134 @@
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Conformance;
/// <summary>Raw-read behaviour over the fixture tag A (see <see cref="ConformanceData.SeedAAsync"/>).</summary>
public abstract class RawReadConformance : ConformanceBase
{
private static readonly CancellationToken Ct = CancellationToken.None;
private static double?[] Values(HistoryReadResult r) => r.Samples.Select(s => (double?)s.Value).ToArray();
[Fact]
public async Task Forward_read_is_start_inclusive_end_exclusive_ascending_with_exact_ticks()
{
await ConformanceData.SeedAAsync(B);
var r = await B.CreateDataSource().ReadRawAsync("A", At(10), At(40), 0, Ct);
Values(r).ShouldBe([20d, 30d, 40d]); // +10 included, +40 (== end) excluded
r.Samples.Select(s => s.SourceTimestampUtc!.Value.Ticks).ShouldBe([B.Round(At(10)).Ticks, B.Round(At(20)).Ticks, B.Round(At(30)).Ticks]);
r.Samples.Select(s => s.StatusCode).ShouldBe([Good, Bad, Good]);
r.Samples[0].SourceTimestampUtc!.Value.Kind.ShouldBe(DateTimeKind.Utc);
r.Samples[0].ServerTimestampUtc.Ticks.ShouldBe(B.Round(At(11)).Ticks); // stored ServerTimeUtc, not "now"
r.ContinuationPoint.ShouldBeNull();
}
[Fact]
public async Task Forward_read_honours_the_limit()
{
await ConformanceData.SeedAAsync(B);
Values(await B.CreateDataSource().ReadRawAsync("A", At(0), At(100), 2, Ct)).ShouldBe([10d, 20d]);
}
[Fact]
public async Task Zero_max_values_is_unbounded()
{
await ConformanceData.SeedAAsync(B);
Values(await B.CreateDataSource().ReadRawAsync("A", At(-1), At(100), 0, Ct)).ShouldBe([10d, 20d, 30d, 40d, 50d]);
Values(await B.CreateDataSource().ReadRawAsync("A", At(-1), At(100), uint.MaxValue, Ct)).Length.ShouldBe(5);
}
[Fact]
public async Task Reverse_read_is_descending_over_end_exclusive_start_inclusive()
{
await ConformanceData.SeedAAsync(B);
// start=+30, end=+10 ⇒ (+10, +30] descending.
Values(await B.CreateDataSource().ReadRawAsync("A", At(30), At(10), 0, Ct)).ShouldBe([40d, 30d]);
Values(await B.CreateDataSource().ReadRawAsync("A", At(100), At(-1), 2, Ct)).ShouldBe([50d, 40d]);
}
[Fact]
public async Task Equal_start_and_end_is_a_point_read()
{
await ConformanceData.SeedAAsync(B);
Values(await B.CreateDataSource().ReadRawAsync("A", At(10), At(10), 0, Ct)).ShouldBe([20d]);
// One resolution unit later: the stored sample is no longer at that instant.
var later = At(10).AddTicks(B.ResolutionTicks);
(await B.CreateDataSource().ReadRawAsync("A", later, later, 0, Ct)).Samples.ShouldBeEmpty();
}
[Fact]
public async Task A_resumed_page_from_the_last_timestamp_inclusive_continues_without_gaps()
{
// The node manager's paging: read cap N, resume at the last returned timestamp INCLUSIVE, trim it.
await ConformanceData.SeedAAsync(B);
var src = B.CreateDataSource();
var p1 = await src.ReadRawAsync("A", At(0), At(100), 2, Ct);
var p2 = await src.ReadRawAsync("A", p1.Samples[^1].SourceTimestampUtc!.Value, At(100), 2, Ct);
Values(p1).ShouldBe([10d, 20d]);
Values(p2).ShouldBe([20d, 30d]); // first is the boundary sample the node manager trims
}
[Fact]
public async Task Unknown_tag_is_an_empty_result_and_reads_never_create_tags()
{
var src = B.CreateDataSource();
(await src.ReadRawAsync("nope", At(0), At(10), 0, Ct)).Samples.ShouldBeEmpty();
await src.ReadAtTimeAsync("nope", [At(1)], Ct);
await src.ReadProcessedAsync("nope", At(0), At(10), TimeSpan.FromSeconds(5), HistoryAggregateType.Average, Ct);
(await B.TagExistsAsync("nope")).ShouldBeFalse();
src.GetHealthSnapshot().TotalFailures.ShouldBe(0);
}
[Fact]
public async Task Outage_markers_and_bad_samples_come_back_with_their_bad_status()
{
await B.SeedAsync("M",
(At(0), HistorianValue.FromDouble(10), Good),
(At(5), HistorianValue.Empty, BadNoCommunication),
(At(7), HistorianValue.FromDouble(99), Bad));
var r = await B.CreateDataSource().ReadRawAsync("M", At(0), At(10), 0, Ct);
r.Samples.Count.ShouldBe(3);
r.Samples[1].Value.ShouldBeNull();
r.Samples[1].StatusCode.ShouldBe(0x80310000u);
r.Samples[2].Value.ShouldBe(99d);
r.Samples[2].StatusCode.ShouldBe(0x80000000u);
}
[Fact]
public async Task Every_kind_round_trips_to_its_CLR_type()
{
var dt = new DateTime(2025, 1, 2, 3, 4, 5, DateTimeKind.Utc).AddTicks(7 * B.ResolutionTicks);
await B.SeedAsync("K",
(At(0), HistorianValue.FromInt64(-9_000_000_000L), Good),
(At(1), HistorianValue.FromBoolean(true), Good),
(At(2), HistorianValue.FromString("héllo"), Good),
(At(3), HistorianValue.FromDateTime(dt), Good),
(At(4), HistorianValue.FromDouble(1.5), Uncertain));
var s = (await B.CreateDataSource().ReadRawAsync("K", At(0), At(5), 0, Ct)).Samples;
s[0].Value.ShouldBe(-9_000_000_000L);
s[1].Value.ShouldBe(true);
s[2].Value.ShouldBe("héllo");
s[3].Value.ShouldBe(dt);
((DateTime)s[3].Value!).Kind.ShouldBe(DateTimeKind.Utc);
s[4].Value.ShouldBe(1.5);
s[4].StatusCode.ShouldBe(0x40000000u);
}
[Fact]
public async Task Successful_reads_are_counted_in_health()
{
await ConformanceData.SeedAAsync(B);
var src = B.CreateDataSource();
await src.ReadRawAsync("A", At(0), At(10), 0, Ct);
await src.ReadRawAsync("A", At(0), At(10), 0, Ct);
var h = src.GetHealthSnapshot();
h.TotalQueries.ShouldBe(2);
h.TotalSuccesses.ShouldBe(2);
h.ConsecutiveFailures.ShouldBe(0);
h.LastSuccessTime.ShouldNotBeNull();
h.ProcessConnectionOpen.ShouldBeTrue();
}
}
tests/Drivers/Historian.Conformance/TagNameConformance.cs
+64
View File
@@ -0,0 +1,64 @@
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Conformance;
/// <summary>
/// Names longer than the 400-character tag-name limit are refused, never truncated. Each test seeds a series under
/// the 400-character PREFIX of the long name: a sized parameter would silently cut the long name to that prefix
/// and hit the wrong series.
/// </summary>
public abstract class TagNameConformance : ConformanceBase
{
private static readonly CancellationToken Ct = CancellationToken.None;
private const uint BadInvalidArgument = 0x80AB0000u;
private const uint BadNoData = 0x809B0000u;
private static readonly string Prefix = new('p', 400);
private static readonly string Long = Prefix + "-x";
[Fact]
public async Task The_writer_refuses_a_long_name_as_a_per_tag_failure_and_writes_nothing()
{
await B.SeedAsync(Prefix, (At(0), HistorianValue.FromDouble(1), Good));
var w = B.CreateValueWriter();
(await w.WriteLiveValuesAsync(Long, [new(At(5), HistorianValue.FromDouble(2), 0u)], Ct)).ShouldBeFalse();
(await B.ReadBackAsync(Prefix)).Count.ShouldBe(1); // nothing landed on the prefix series
(await B.TagExistsAsync(Long)).ShouldBeFalse(); // and the long tag was not created
}
[Fact]
public async Task The_updater_answers_BadInvalidArgument_for_every_item_of_a_long_name()
{
await B.SeedAsync(Prefix, (At(0), HistorianValue.FromDouble(1), Good));
var u = B.CreateUpdater();
(await u.UpdateDataAsync(Long, HistoryUpdateMode.Update,
[new(At(0), HistorianValue.FromDouble(9), 0u), new(At(1), HistorianValue.FromDouble(9), 0u)], Ct))
.ShouldBe([BadInvalidArgument, BadInvalidArgument]);
(await u.DeleteRawAsync(Long, At(-1), At(10), Ct)).ShouldBe(BadInvalidArgument);
(await u.DeleteAtTimeAsync(Long, [At(0)], Ct)).ShouldBe([BadInvalidArgument]);
// The prefix series is untouched: not replaced, not deleted.
var raw = (await B.CreateDataSource().ReadRawAsync(Prefix, At(-1), At(10), 0, Ct)).Samples;
raw.Select(s => s.Value).ShouldBe([(object)1d]);
}
[Fact]
public async Task Reads_of_a_long_name_are_not_found_rather_than_the_prefix_series()
{
await B.SeedAsync(Prefix, (At(0), HistorianValue.FromDouble(1), Good));
var src = B.CreateDataSource();
(await src.ReadRawAsync(Long, At(-1), At(10), 0, Ct)).Samples.ShouldBeEmpty();
(await src.ReadAtTimeAsync(Long, [At(0)], Ct)).Samples.Single().StatusCode.ShouldBe(BadNoData);
(await src.ReadProcessedAsync(Long, At(-1), At(10), TimeSpan.Zero, HistoryAggregateType.Count, Ct))
.Samples.Single().StatusCode.ShouldBe(HistorianInfoBits.WithCalculated(BadNoData));
src.GetHealthSnapshot().TotalFailures.ShouldBe(0);
}
}
tests/Drivers/Historian.Conformance/UpdateConformance.cs
+173
View File
@@ -0,0 +1,173 @@
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Conformance;
/// <summary>History update/delete result codes and their effect on storage.</summary>
public abstract class UpdateConformance : ConformanceBase
{
private const uint UncertainStatus = HistorianStatusCodes.UncertainDataSubNormal;
// Deliberately NOT millisecond (or microsecond) aligned: sub-resolution ticks must survive; stored times are B.Round(T(n)).
private static readonly DateTime T0 = new(2026, 1, 1, 0, 0, 0, DateTimeKind.Utc);
private static DateTime T(int n) => T0.AddTicks(1234567L * n + 3);
private static HistorizationValue V(DateTime? t, double d, uint status = Good) =>
new(t, HistorianValue.FromDouble(d), status);
private static readonly CancellationToken Ct = CancellationToken.None;
[Fact]
public async Task Insert_new_is_GoodEntryInserted_and_existing_is_BadEntryExists_in_order()
{
var u = B.CreateUpdater();
var r1 = await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T(1), 1), V(T(2), 2)], Ct);
r1.ShouldBe([HistorianStatusCodes.GoodEntryInserted, HistorianStatusCodes.GoodEntryInserted]);
// Mixed: T(2) exists, T(3) new, T(1) exists - out of timestamp order on purpose.
var r2 = await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T(3), 3), V(T(2), 99), V(T(1), 98)], Ct);
r2.ShouldBe([HistorianStatusCodes.GoodEntryInserted, HistorianStatusCodes.BadEntryExists, HistorianStatusCodes.BadEntryExists]);
var rows = await B.ReadBackAsync("t");
rows.Select(x => (double?)x.V.Number).ShouldBe([1d, 2d, 3d]); // existing values untouched
rows.Select(x => x.T).ShouldBe([B.Round(T(1)), B.Round(T(2)), B.Round(T(3))]); // sub-ms ticks preserved
}
[Fact]
public async Task Replace_existing_is_GoodEntryReplaced_with_new_value_and_status_missing_is_BadNoEntryExists()
{
var u = B.CreateUpdater();
await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T(1), 1), V(T(2), 2)], Ct);
var r = await u.UpdateDataAsync("t", HistoryUpdateMode.Replace,
[V(T(2), 22, UncertainStatus), V(T(5), 5), V(T(1), 11)], Ct);
r.ShouldBe([HistorianStatusCodes.GoodEntryReplaced, HistorianStatusCodes.BadNoEntryExists, HistorianStatusCodes.GoodEntryReplaced]);
var rows = await B.ReadBackAsync("t");
rows.Count.ShouldBe(2); // T(5) was not created
rows[0].V.Number.ShouldBe(11d);
rows[0].Status.ShouldBe(Good);
rows[1].V.Number.ShouldBe(22d);
rows[1].Status.ShouldBe(UncertainStatus); // replaced value visible with its new status
}
[Fact]
public async Task Update_upserts_inserted_or_replaced_per_item()
{
var u = B.CreateUpdater();
await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T(1), 1)], Ct);
var r = await u.UpdateDataAsync("t", HistoryUpdateMode.Update, [V(T(2), 2), V(T(1), 10)], Ct);
r.ShouldBe([HistorianStatusCodes.GoodEntryInserted, HistorianStatusCodes.GoodEntryReplaced]);
(await B.ReadBackAsync("t")).Select(x => (double?)x.V.Number).ShouldBe([10d, 2d]);
}
[Fact]
public async Task Unknown_tag_Insert_and_Update_create_it_but_Replace_and_deletes_do_not()
{
var u = B.CreateUpdater();
(await u.UpdateDataAsync("rep", HistoryUpdateMode.Replace, [V(T(1), 1)], Ct))
.ShouldBe([HistorianStatusCodes.BadNoEntryExists]);
(await u.DeleteRawAsync("del", T(0), T(9), Ct)).ShouldBe(HistorianStatusCodes.BadNoData);
(await u.DeleteAtTimeAsync("del2", [T(1)], Ct)).ShouldBe([HistorianStatusCodes.BadNoEntryExists]);
foreach (var name in new[] { "rep", "del", "del2" })
(await B.TagExistsAsync(name)).ShouldBeFalse(name); // none of those created a tag
(await u.UpdateDataAsync("ins", HistoryUpdateMode.Insert, [V(T(1), 1)], Ct)).ShouldBe([HistorianStatusCodes.GoodEntryInserted]);
(await u.UpdateDataAsync("upd", HistoryUpdateMode.Update, [V(T(1), 1)], Ct)).ShouldBe([HistorianStatusCodes.GoodEntryInserted]);
(await B.TagExistsAsync("ins")).ShouldBeTrue();
(await B.TagExistsAsync("upd")).ShouldBeTrue();
}
[Fact]
public async Task Null_and_MinValue_timestamps_are_BadInvalidTimestamp_and_others_proceed()
{
var u = B.CreateUpdater();
var r = await u.UpdateDataAsync("t", HistoryUpdateMode.Insert,
[V(null, 1), V(T(1), 2), V(DateTime.MinValue, 3)], Ct);
r.ShouldBe([HistorianStatusCodes.BadInvalidTimestamp, HistorianStatusCodes.GoodEntryInserted, HistorianStatusCodes.BadInvalidTimestamp]);
(await B.ReadBackAsync("t")).Count.ShouldBe(1);
// All-invalid request: no DB work, no tag created, still one code per item.
var u2 = B.CreateUpdater();
await u2.UpdateDataAsync("seed", HistoryUpdateMode.Insert, [V(T(1), 1)], Ct); // bootstraps the schema
(await u2.UpdateDataAsync("x", HistoryUpdateMode.Insert, [V(null, 1)], Ct)).ShouldBe([HistorianStatusCodes.BadInvalidTimestamp]);
(await B.TagExistsAsync("seed")).ShouldBeTrue();
(await B.TagExistsAsync("x")).ShouldBeFalse(); // "x" was never created
}
[Fact]
public async Task NaN_and_Infinity_are_rejected_with_BadInvalidArgument_and_others_proceed()
{
var u = B.CreateUpdater();
var r = await u.UpdateDataAsync("t", HistoryUpdateMode.Insert,
[V(T(1), double.NaN), V(T(2), 2), V(T(3), double.PositiveInfinity), V(T(4), double.NegativeInfinity)], Ct);
r.ShouldBe([HistorianStatusCodes.BadInvalidArgument, HistorianStatusCodes.GoodEntryInserted,
HistorianStatusCodes.BadInvalidArgument, HistorianStatusCodes.BadInvalidArgument]);
(await B.ReadBackAsync("t")).Select(x => (double?)x.V.Number).ShouldBe([2d]);
}
[Fact]
public async Task Duplicate_timestamp_in_one_Insert_request_second_is_BadEntryExists_first_wins()
{
var u = B.CreateUpdater();
var r = await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T(1), 1), V(T(2), 2), V(T(1), 99)], Ct);
r.ShouldBe([HistorianStatusCodes.GoodEntryInserted, HistorianStatusCodes.GoodEntryInserted, HistorianStatusCodes.BadEntryExists]);
(await B.ReadBackAsync("t")).Select(x => (double?)x.V.Number).ShouldBe([1d, 2d]);
}
[Fact]
public async Task Duplicate_timestamp_in_one_Replace_or_Update_request_last_wins_and_all_share_the_code()
{
var u = B.CreateUpdater();
await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T(1), 1)], Ct);
var rep = await u.UpdateDataAsync("t", HistoryUpdateMode.Replace, [V(T(1), 5), V(T(1), 6)], Ct);
rep.ShouldBe([HistorianStatusCodes.GoodEntryReplaced, HistorianStatusCodes.GoodEntryReplaced]);
(await B.ReadBackAsync("t")).Single().V.Number.ShouldBe(6d);
var upd = await u.UpdateDataAsync("t", HistoryUpdateMode.Update, [V(T(7), 7), V(T(7), 8)], Ct);
upd.ShouldBe([HistorianStatusCodes.GoodEntryInserted, HistorianStatusCodes.GoodEntryInserted]);
(await B.ReadBackAsync("t")).Last().V.Number.ShouldBe(8d);
}
[Fact]
public async Task Replace_and_Update_can_change_the_value_kind_and_clear_stale_columns()
{
var u = B.CreateUpdater();
await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [new HistorizationValue(T(1), HistorianValue.FromDouble(1.5), Good)], Ct);
(await u.UpdateDataAsync("t", HistoryUpdateMode.Replace,
[new HistorizationValue(T(1), HistorianValue.FromString("hi"), Good)], Ct)).ShouldBe([HistorianStatusCodes.GoodEntryReplaced]);
var rows = await B.ReadBackAsync("t");
rows.Single().V.ShouldBe(HistorianValue.FromString("hi")); // Number cleared (0), not left at 1.5
}
[Fact]
public async Task DeleteRaw_is_half_open_Good_when_rows_removed_BadNoData_when_none()
{
var u = B.CreateUpdater();
await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T(1), 1), V(T(2), 2), V(T(3), 3)], Ct);
// [T(1), T(3)) removes T(1) and T(2); T(3) (the end) survives.
(await u.DeleteRawAsync("t", T(1), T(3), Ct)).ShouldBe(Good);
(await B.ReadBackAsync("t")).Select(x => x.T).ShouldBe([B.Round(T(3))]);
(await u.DeleteRawAsync("t", T(1), T(3), Ct)).ShouldBe(HistorianStatusCodes.BadNoData);
// sub-ms boundary: a start one resolution unit after T(3) excludes it.
(await u.DeleteRawAsync("t", T(3).AddTicks(B.ResolutionTicks), T(3).AddTicks(100 * B.ResolutionTicks), Ct)).ShouldBe(HistorianStatusCodes.BadNoData);
(await B.ReadBackAsync("t")).Count.ShouldBe(1);
}
[Fact]
public async Task DeleteAtTime_reports_per_time_in_order_with_exact_tick_matching()
{
var u = B.CreateUpdater();
await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T(1), 1), V(T(2), 2), V(T(3), 3)], Ct);
var r = await u.DeleteAtTimeAsync("t", [T(3), T(2).AddTicks(B.ResolutionTicks) /* near miss */, T(1), DateTime.MinValue], Ct);
r.ShouldBe([Good, HistorianStatusCodes.BadNoEntryExists, Good, HistorianStatusCodes.BadInvalidTimestamp]);
(await B.ReadBackAsync("t")).Select(x => x.T).ShouldBe([B.Round(T(2))]);
}
[Fact]
public async Task DeleteAtTime_duplicate_time_first_reports_Good_later_BadNoEntryExists()
{
var u = B.CreateUpdater();
await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T(1), 1)], Ct);
(await u.DeleteAtTimeAsync("t", [T(1), T(1)], Ct)).ShouldBe([Good, HistorianStatusCodes.BadNoEntryExists]);
(await B.ReadBackAsync("t")).ShouldBeEmpty();
}
}
tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/AlarmWriterTests.cs
+118
View File
@@ -0,0 +1,118 @@
using Npgsql;
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian;
using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests;
[Collection(QuestDbCollection.Name)]
public sealed class AlarmWriterTests(QuestDbFixture fx)
{
// µs-aligned: QuestDB stores µs, so the ts column reads back exactly.
private static readonly DateTime T = new DateTime(2026, 10, 9, 1, 2, 3, DateTimeKind.Utc).AddTicks(1230);
private (QuestDbAlarmHistorianWriter W, QuestDbConnection C) Make(string? connection = null, bool auto = true)
{
var o = fx.Options(fx.UniquePrefix(), autoCreate: auto);
if (connection is not null) o.ConnectionString = connection;
var c = new QuestDbConnection(o);
return (new QuestDbAlarmHistorianWriter(c, new QuestDbSchemaBootstrapper(c, o), new QuestDbWalVisibility(c, o.VisibilityTimeout)), c);
}
private static AlarmHistorianEvent Ev(string kind = "Activated", string type = "LimitAlarm", string? comment = "c", DateTime? at = null) =>
new("alarm-1", "Plant/Line/Eq", "HiTemp", type, AlarmSeverity.High, kind, "msg", "user", comment, at ?? T);
private static async Task<long> CountAsync(QuestDbConnection c, string? kind = null)
{
await using var conn = await c.OpenAsync(default);
await using var cmd = c.CreateCommand(conn, $"SELECT count() FROM {c.EventTable}" + (kind is null ? "" : " WHERE event_kind = $1"));
if (kind is not null) QuestDbConnection.BindText(cmd, kind);
return Convert.ToInt64(await cmd.ExecuteScalarAsync());
}
[Fact]
public async Task Event_round_trips_and_a_repeat_is_one_row_with_every_write_Ack()
{
fx.SkipIfUnavailable();
var (w, c) = Make();
(await w.WriteBatchAsync([Ev(), Ev()], CancellationToken.None)).ShouldBe([HistorianWriteOutcome.Ack, HistorianWriteOutcome.Ack]);
(await w.WriteBatchAsync([Ev()], CancellationToken.None)).ShouldBe([HistorianWriteOutcome.Ack]);
(await w.WriteBatchAsync([Ev(kind: "Cleared")], CancellationToken.None)).ShouldBe([HistorianWriteOutcome.Ack]);
await using var conn = await c.OpenAsync(default);
await using var cmd = c.CreateCommand(conn,
$"SELECT event_id, source, alarm_id, alarm_name, alarm_type_name, severity, message, user_name, comment, ts, received_utc FROM {c.EventTable} WHERE event_kind = 'Activated'");
await using var r = await cmd.ExecuteReaderAsync();
(await r.ReadAsync()).ShouldBeTrue();
r.GetString(0).ShouldBe(AlarmEventIds.ComputeEventId(Ev()));
r.GetString(1).ShouldBe("Plant/Line/Eq");
r.GetString(2).ShouldBe("alarm-1");
r.GetString(3).ShouldBe("HiTemp");
r.GetString(4).ShouldBe("LimitAlarm");
r.GetInt32(5).ShouldBe(AlarmEventIds.ToNumericSeverity(AlarmSeverity.High));
r.GetInt32(5).ShouldBe(700);
r.GetString(6).ShouldBe("msg");
r.GetString(7).ShouldBe("user");
r.GetString(8).ShouldBe("c");
QuestDbConnection.ReadUtc(r, 9).ShouldBe(T);
(DateTime.UtcNow - QuestDbConnection.ReadUtc(r, 10)).ShouldBeLessThan(TimeSpan.FromMinutes(1));
(await r.ReadAsync()).ShouldBeFalse();
await r.CloseAsync();
(await CountAsync(c)).ShouldBe(2);
}
[Fact]
public async Task A_null_comment_and_a_very_long_type_name_are_both_stored_and_Ack()
{
fx.SkipIfUnavailable();
var (w, c) = Make();
var r = await w.WriteBatchAsync([Ev(kind: "A"), Ev(kind: "B", type: new string('t', 500)), Ev(kind: "C", comment: null)], CancellationToken.None);
r.ShouldBe([HistorianWriteOutcome.Ack, HistorianWriteOutcome.Ack, HistorianWriteOutcome.Ack]);
(await CountAsync(c)).ShouldBe(3);
}
[Fact]
public async Task An_event_before_1970_is_PermanentFail_and_siblings_still_Ack()
{
fx.SkipIfUnavailable();
var (w, c) = Make();
var r = await w.WriteBatchAsync([Ev(kind: "A"), Ev(kind: "B", at: new DateTime(1960, 1, 1, 0, 0, 0, DateTimeKind.Utc)), Ev(kind: "C")], CancellationToken.None);
r.ShouldBe([HistorianWriteOutcome.Ack, HistorianWriteOutcome.PermanentFail, HistorianWriteOutcome.Ack]);
(await CountAsync(c)).ShouldBe(2);
}
[Fact]
public async Task Concurrent_writers_of_the_same_event_all_Ack_with_one_row()
{
fx.SkipIfUnavailable();
var (w, c) = Make();
await w.WriteBatchAsync([Ev(kind: "warm")], CancellationToken.None);
var rs = await Task.WhenAll(Enumerable.Range(0, 12).Select(_ => Task.Run(() => w.WriteBatchAsync([Ev()], CancellationToken.None))));
rs.ShouldAllBe(x => x[0] == HistorianWriteOutcome.Ack);
(await CountAsync(c, "Activated")).ShouldBe(1);
}
[Fact]
public async Task Unreachable_database_is_RetryPlease_not_an_exception_and_empty_batch_is_empty()
{
fx.SkipIfUnavailable();
var (w, _) = Make("Host=localhost;Port=1;Username=admin;Password=quest;Database=qdb;ServerCompatibilityMode=NoTypeLoading;Timeout=2", auto: false);
(await w.WriteBatchAsync([Ev()], CancellationToken.None)).ShouldBe([HistorianWriteOutcome.RetryPlease]);
(await w.WriteBatchAsync([], CancellationToken.None)).ShouldBeEmpty();
}
[Fact]
public void Classify_maps_unknown_and_transport_errors_to_RetryPlease_and_data_rejections_to_PermanentFail()
{
QuestDbAlarmHistorianWriter.Classify(new InvalidOperationException("boom")).ShouldBe(HistorianWriteOutcome.RetryPlease);
QuestDbAlarmHistorianWriter.Classify(new IOException("reset")).ShouldBe(HistorianWriteOutcome.RetryPlease);
QuestDbAlarmHistorianWriter.Classify(new TimeoutException()).ShouldBe(HistorianWriteOutcome.RetryPlease);
QuestDbAlarmHistorianWriter.Classify(new PostgresException("table is busy", "ERROR", "ERROR", "00000")).ShouldBe(HistorianWriteOutcome.RetryPlease);
QuestDbAlarmHistorianWriter.Classify(new PostgresException("designated timestamp before 1970-01-01 is not allowed", "ERROR", "ERROR", "00000"))
.ShouldBe(HistorianWriteOutcome.PermanentFail);
QuestDbAlarmHistorianWriter.Classify(new PostgresException("inconvertible value: x [VARCHAR -> INT]", "ERROR", "ERROR", "00000"))
.ShouldBe(HistorianWriteOutcome.PermanentFail);
}
}
tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/AtTimeRegressionTests.cs
+85
View File
@@ -0,0 +1,85 @@
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.Driver.Historian.Conformance;
using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests;
/// <summary>
/// QuestDB AtTime regressions found in the T5 review (M1, M3), and the data source's reported timestamp
/// resolution (the HistoryRead paging step, C1).
/// </summary>
[Collection(QuestDbCollection.Name)]
public sealed class AtTimeRegressionTests(QuestDbFixture fx) : IAsyncLifetime
{
private static readonly CancellationToken Ct = CancellationToken.None;
private const uint GoodInterpolated = 0x00000402u;
private QuestDbConformanceBackend _b = null!;
private static DateTime At(double s) => ConformanceData.At(s);
public async ValueTask InitializeAsync() => _b = await QuestDbConformanceBackend.CreateConcreteAsync(fx);
public async ValueTask DisposeAsync()
{
if (_b is not null) await _b.DisposeAsync();
}
/// <summary>
/// M3: a request time with a sub-microsecond remainder is answered stamped with THAT time (as SQL Server does),
/// not with the rounded time the seek used. The value is still the one at the rounded time.
/// </summary>
[Fact]
public async Task AtTime_result_is_stamped_with_the_original_sub_microsecond_request_time()
{
await ConformanceData.SeedAAsync(_b); // +0=10, +10=20, ...
var requested = _b.Round(At(5)).AddTicks(3); // 0.3 µs past a microsecond (T0 itself is not µs-aligned)
var unspecified = DateTime.SpecifyKind(_b.Round(At(7)).AddTicks(7), DateTimeKind.Unspecified);
var s = (await _b.CreateDataSource().ReadAtTimeAsync("A", [requested, unspecified], Ct)).Samples;
s[0].SourceTimestampUtc.ShouldBe(requested);
s[0].SourceTimestampUtc!.Value.Kind.ShouldBe(DateTimeKind.Utc);
s[0].Value.ShouldBe(15d); // evaluated at the rounded time Round(At(5))
s[0].StatusCode.ShouldBe(GoodInterpolated);
s[1].SourceTimestampUtc.ShouldBe(DateTime.SpecifyKind(unspecified, DateTimeKind.Utc));
s[1].SourceTimestampUtc!.Value.Ticks.ShouldBe(unspecified.Ticks);
}
/// <summary>M3 on the unknown-tag path: every BadNoData entry carries its own requested time.</summary>
[Fact]
public async Task AtTime_unknown_tag_entries_are_stamped_with_the_original_request_time()
{
var requested = _b.Round(At(5)).AddTicks(3);
var s = (await _b.CreateDataSource().ReadAtTimeAsync("nope", [requested], Ct)).Samples;
s.Single().StatusCode.ShouldBe(HistorianStatusCodes.BadNoData);
s.Single().SourceTimestampUtc.ShouldBe(requested);
}
/// <summary>
/// M1: the NEXT seek skips a tombstone and finds the next live row. With +10 deleted, At(5) lies between the
/// live +0 (10) and the live +20 (30), so it interpolates to 15, Good. Seeing the tombstone as "next" would
/// instead hold 10 as Uncertain (an Empty BadNoData neighbour is not usable).
/// </summary>
[Fact]
public async Task AtTime_next_seek_skips_a_tombstone_and_finds_the_next_live_row()
{
await _b.SeedAsync("M",
(At(0), HistorianValue.FromDouble(10), ConformanceData.Good),
(At(10), HistorianValue.FromDouble(20), ConformanceData.Good),
(At(20), HistorianValue.FromDouble(30), ConformanceData.Good));
await _b.InsertRowsAsync("M", [(At(10), HistorianValue.Empty, HistorianStatusCodes.BadNoData, true)]);
var s = (await _b.CreateDataSource().ReadAtTimeAsync("M", [At(5)], Ct)).Samples.Single();
s.Value.ShouldBe(15d); // 10 + (30 - 10) * 5/20
s.StatusCode.ShouldBe(GoodInterpolated);
}
/// <summary>C1: the data source reports QuestDB's microsecond storage resolution to the paging code.</summary>
[Fact]
public void Data_source_reports_a_ten_tick_timestamp_resolution() =>
((IHistorianDataSource)_b.CreateDataSource()).TimestampResolutionTicks.ShouldBe(10);
}
tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/BootstrapTests.cs
+241
View File
@@ -0,0 +1,241 @@
using Npgsql;
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests;
[Collection(QuestDbCollection.Name)]
public sealed class BootstrapTests(QuestDbFixture fx)
{
private (QuestDbConnection Conn, QuestDbSchemaBootstrapper Boot, QuestDbHistorianOptions Opts) Make(
string prefix, int? retention = null, bool autoCreate = true)
{
var o = fx.Options(prefix, retention, autoCreate);
var c = new QuestDbConnection(o);
return (c, new QuestDbSchemaBootstrapper(c, o), o);
}
private static async Task<List<string>> TablesAsync(QuestDbConnection c, string prefix)
{
await using var conn = await c.OpenAsync(default);
await using var cmd = c.CreateCommand(conn, "SELECT table_name FROM tables() WHERE starts_with(table_name, $1) ORDER BY table_name");
QuestDbConnection.BindText(cmd, prefix);
var names = new List<string>();
await using var r = await cmd.ExecuteReaderAsync();
while (await r.ReadAsync()) names.Add(r.GetString(0));
return names;
}
private static async Task<(long Value, string Unit)> TtlAsync(QuestDbConnection c, string table)
{
await using var conn = await c.OpenAsync(default);
await using var cmd = c.CreateCommand(conn, "SELECT ttlValue, ttlUnit FROM tables() WHERE table_name = $1");
QuestDbConnection.BindText(cmd, table);
await using var r = await cmd.ExecuteReaderAsync();
(await r.ReadAsync()).ShouldBeTrue();
return (Convert.ToInt64(r.GetValue(0)), r.GetString(1));
}
private static async Task<long> SequencerTxnAsync(QuestDbConnection c, string table)
{
await using var conn = await c.OpenAsync(default);
await using var cmd = c.CreateCommand(conn, "SELECT sequencerTxn FROM wal_tables() WHERE name = $1");
QuestDbConnection.BindText(cmd, table);
return Convert.ToInt64(await cmd.ExecuteScalarAsync());
}
[Fact]
public async Task Tables_are_created()
{
fx.SkipIfUnavailable();
var p = fx.UniquePrefix();
var (c, boot, _) = Make(p);
await boot.EnsureAsync(default);
(await TablesAsync(c, p)).ShouldBe([p + "event", p + "sample", p + "tag"]);
}
[Fact]
public async Task A_second_run_is_a_noop_and_issues_no_alter()
{
fx.SkipIfUnavailable();
var p = fx.UniquePrefix();
var (c, boot, _) = Make(p, retention: 7);
await boot.EnsureAsync(default);
var before = await SequencerTxnAsync(c, p + "sample");
var (c2, boot2, _) = Make(p, retention: 7); // a fresh instance on a table whose TTL already matches (stored as 1 WEEK)
await boot2.EnsureAsync(default);
await Task.Delay(300); // an ALTER would have been applied by now
(await SequencerTxnAsync(c2, p + "sample")).ShouldBe(before);
(await SequencerTxnAsync(c2, p + "event")).ShouldBe(await SequencerTxnAsync(c, p + "event"));
(await TablesAsync(c2, p)).Count.ShouldBe(3);
}
[Fact]
public async Task Two_concurrent_bootstrappers_both_succeed()
{
fx.SkipIfUnavailable();
for (var round = 0; round < 5; round++)
{
var p = fx.UniquePrefix();
var a = Make(p, retention: 7);
var b = Make(p, retention: 7);
await Task.WhenAll(
Task.Run(() => a.Boot.EnsureAsync(default)),
Task.Run(() => b.Boot.EnsureAsync(default)));
(await TablesAsync(a.Conn, p)).Count.ShouldBe(3);
}
}
[Fact]
public async Task Ttl_is_in_metadata_when_retention_is_7_days_and_absent_when_unset()
{
fx.SkipIfUnavailable();
var withTtl = fx.UniquePrefix();
var (c1, b1, _) = Make(withTtl, retention: 7);
await b1.EnsureAsync(default);
QuestDbTables.TtlHours((await TtlAsync(c1, withTtl + "sample")).Value, (await TtlAsync(c1, withTtl + "sample")).Unit).ShouldBe(168);
QuestDbTables.TtlHours((await TtlAsync(c1, withTtl + "event")).Value, (await TtlAsync(c1, withTtl + "event")).Unit).ShouldBe(168);
(await TtlAsync(c1, withTtl + "tag")).Value.ShouldBe(0); // the tag catalog never expires
var noTtl = fx.UniquePrefix();
var (c2, b2, _) = Make(noTtl);
await b2.EnsureAsync(default);
(await TtlAsync(c2, noTtl + "sample")).Value.ShouldBe(0);
(await TtlAsync(c2, noTtl + "event")).Value.ShouldBe(0);
}
[Fact]
public async Task A_changed_retention_is_reapplied_and_removal_clears_it()
{
fx.SkipIfUnavailable();
var p = fx.UniquePrefix();
var (c, b7, _) = Make(p, retention: 7);
await b7.EnsureAsync(default);
await Make(p, retention: 30).Boot.EnsureAsync(default);
await AwaitHoursAsync(c, p + "sample", 720);
await AwaitHoursAsync(c, p + "event", 720);
await Make(p).Boot.EnsureAsync(default); // unset => SET TTL 0
await AwaitHoursAsync(c, p + "sample", 0);
}
private static async Task AwaitHoursAsync(QuestDbConnection c, string table, long hours)
{
var deadline = DateTime.UtcNow.AddSeconds(10);
long got;
do
{
var (v, u) = await TtlAsync(c, table);
got = QuestDbTables.TtlHours(v, u);
if (got == hours) return;
await Task.Delay(50); // the ALTER is a WAL transaction: visible only once applied
} while (DateTime.UtcNow < deadline);
got.ShouldBe(hours);
}
[Fact]
public async Task AutoCreateSchema_false_against_missing_tables_throws_HistorianSchemaMissingException()
{
fx.SkipIfUnavailable();
var (_, boot, _) = Make(fx.UniquePrefix(), autoCreate: false);
var ex = await Should.ThrowAsync<HistorianSchemaMissingException>(() => boot.EnsureAsync(default));
ex.Message.ShouldContain("AutoCreateSchema");
}
[Fact]
public async Task AutoCreateSchema_false_passes_once_the_tables_exist()
{
fx.SkipIfUnavailable();
var p = fx.UniquePrefix();
await Make(p).Boot.EnsureAsync(default);
await Make(p, autoCreate: false).Boot.EnsureAsync(default);
}
[Fact]
public async Task ExistsAsync_is_false_for_an_unknown_tag_and_true_after_EnsureAsync()
{
fx.SkipIfUnavailable();
var p = fx.UniquePrefix();
var (c, boot, o) = Make(p);
var catalog = new QuestDbTagCatalog(c, boot, new QuestDbWalVisibility(c, o.VisibilityTimeout));
(await catalog.ExistsAsync("Plant.Pump1.Speed", default)).ShouldBeFalse();
await catalog.EnsureAsync("Plant.Pump1.Speed", 3, default);
(await catalog.ExistsAsync("Plant.Pump1.Speed", default)).ShouldBeTrue();
// A fresh catalog (no cache) sees it through the table, and a different name is still unknown.
var fresh = new QuestDbTagCatalog(c, boot, new QuestDbWalVisibility(c, o.VisibilityTimeout));
(await fresh.ExistsAsync("Plant.Pump1.Speed", default)).ShouldBeTrue();
(await fresh.ExistsAsync("Plant.Pump1.Other", default)).ShouldBeFalse();
await catalog.EnsureAsync("Plant.Pump1.Speed", 3, default); // idempotent
}
[Fact]
public async Task An_over_long_tag_name_is_unknown_and_cannot_be_ensured()
{
fx.SkipIfUnavailable();
var p = fx.UniquePrefix();
var (c, boot, o) = Make(p);
var catalog = new QuestDbTagCatalog(c, boot, new QuestDbWalVisibility(c, o.VisibilityTimeout));
var name = new string('x', HistorianTagNames.MaxLength + 1);
(await catalog.ExistsAsync(name, default)).ShouldBeFalse();
await Should.ThrowAsync<ArgumentException>(() => catalog.EnsureAsync(name, 3, default));
}
[Fact]
public async Task Wait_returns_once_writerTxn_reaches_the_captured_sequencerTxn()
{
fx.SkipIfUnavailable();
var p = fx.UniquePrefix();
var (c, boot, o) = Make(p);
await boot.EnsureAsync(default);
var wal = new QuestDbWalVisibility(c, TimeSpan.FromSeconds(5));
for (var i = 0; i < 20; i++)
{
var ts = new DateTime(2026, 3, 1, 0, 0, i, DateTimeKind.Utc);
await using (var conn = await c.OpenAsync(default))
{
await using var cmd = c.CreateCommand(conn,
$"INSERT INTO {c.SampleTable} (ts, tag, kind, num_value, status, deleted) VALUES ($1, $2, $3, $4, $5, false)");
QuestDbConnection.BindTimestamp(cmd, ts);
QuestDbConnection.BindText(cmd, "wal-tag");
QuestDbConnection.BindByte(cmd, 3);
cmd.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlTypes.NpgsqlDbType.Double, Value = (double)i });
QuestDbConnection.BindLong(cmd, 0);
await cmd.ExecuteNonQueryAsync();
}
(await wal.WaitAsync(c.SampleTable, default)).ShouldBeTrue();
// After the wait the row is visible on the very next query (the T0 gap was 7/20 without it).
await using var q = await c.OpenAsync(default);
await using var count = c.CreateCommand(q, $"SELECT count() FROM {c.SampleTable} WHERE tag = $1");
QuestDbConnection.BindText(count, "wal-tag");
Convert.ToInt64(await count.ExecuteScalarAsync()).ShouldBe(i + 1);
}
}
[Fact]
public async Task Timeout_zero_skips_the_wait()
{
fx.SkipIfUnavailable();
var p = fx.UniquePrefix();
var (c, boot, _) = Make(p);
await boot.EnsureAsync(default);
var wal = new QuestDbWalVisibility(c, TimeSpan.Zero);
var sw = System.Diagnostics.Stopwatch.StartNew();
(await wal.WaitAsync(c.SampleTable, default)).ShouldBeTrue();
sw.ElapsedMilliseconds.ShouldBeLessThan(50); // no round-trip at all
}
[Fact]
public async Task Waiting_on_a_table_that_is_not_a_WAL_table_returns_immediately()
{
fx.SkipIfUnavailable();
var p = fx.UniquePrefix();
var (c, boot, _) = Make(p);
await boot.EnsureAsync(default);
var wal = new QuestDbWalVisibility(c, TimeSpan.FromSeconds(5));
(await wal.WaitAsync(p + "no_such_table", default)).ShouldBeTrue();
}
}
tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/PagingRegressionTests.cs
+118
View File
@@ -0,0 +1,118 @@
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.Driver.Historian.Conformance;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests;
/// <summary>
/// C1 (T7 review I-1): the facts the server's HistoryRead-Raw paging relies on, pinned against the REAL QuestDB data
/// source (the node-manager walk in <c>NodeManagerHistoryReadPagingTests</c> runs on a fake that models them).
/// <c>HistoryPaging</c> is internal to OpcUaServer and not visible from this project, so the end-to-end walk below
/// re-states the node manager's cursor arithmetic (resume inclusive + boundary trim; stalled ⇒ point-read the
/// cluster and step off by <see cref="IHistorianDataSource.TimestampResolutionTicks"/>) over the real reads.
/// </summary>
[Collection(QuestDbCollection.Name)]
public sealed class PagingRegressionTests(QuestDbFixture fx) : IAsyncLifetime
{
private static readonly CancellationToken Ct = CancellationToken.None;
private static readonly DateTime T0 = new(2026, 3, 1, 12, 0, 0, DateTimeKind.Utc); // µs-aligned
private QuestDbConformanceBackend _b = null!;
/// <summary>Sample i at T0 + i µs (adjacent microseconds: a one-tick step lands strictly between two of them).</summary>
private static DateTime S(int i) => T0.AddTicks(10L * i);
public async ValueTask InitializeAsync()
{
_b = await QuestDbConformanceBackend.CreateConcreteAsync(fx);
await _b.SeedAsync("P", Enumerable.Range(0, 6)
.Select(i => (S(i), HistorianValue.FromDouble(i), ConformanceData.Good)).ToArray());
}
public async ValueTask DisposeAsync()
{
if (_b is not null) await _b.DisposeAsync();
}
private async Task<DateTime?[]> RawAsync(IHistorianDataSource ds, DateTime start, DateTime end, uint max) =>
(await ds.ReadRawAsync("P", start, end, max, Ct)).Samples.Select(s => s.SourceTimestampUtc).ToArray();
[Fact]
public async Task Forward_raw_reads_truncate_one_tick_but_advance_by_one_resolution_step()
{
var ds = _b.CreateDataSource();
var end = T0.AddHours(1);
var res = ds.TimestampResolutionTicks;
(await RawAsync(ds, S(2).AddTicks(1), end, 1)).ShouldBe([S(2)]); // T + 1 tick ⇒ back to T (the C1 loop)
(await RawAsync(ds, S(2).AddTicks(res), end, 1)).ShouldBe([S(3)]); // T + resolution ⇒ the next sample
(await RawAsync(ds, S(2), S(2), 10)).ShouldBe([S(2)]); // point read at T ⇒ T
}
[Fact]
public async Task Reverse_raw_reads_step_down_by_one_resolution_step()
{
var ds = _b.CreateDataSource();
var end = T0.AddHours(-1);
var res = ds.TimestampResolutionTicks;
(await RawAsync(ds, S(3).AddTicks(-1), end, 1)).ShouldBe([S(2)]); // T − 1 tick truncates DOWN ⇒ previous
(await RawAsync(ds, S(3).AddTicks(-res), end, 1)).ShouldBe([S(2)]); // T − resolution ⇒ previous
(await RawAsync(ds, S(3), end, 1)).ShouldBe([S(3)]); // reverse window (end, start] includes start
}
[Theory]
[InlineData(false)]
[InlineData(true)]
public async Task One_value_per_page_walk_returns_every_sample_once_and_terminates(bool reverse)
{
var ds = _b.CreateDataSource();
var (start, end) = reverse ? (T0.AddHours(1), T0.AddHours(-1)) : (T0.AddHours(-1), T0.AddHours(1));
var collected = await WalkAsync(ds, start, end);
var expected = Enumerable.Range(0, 6).Select(i => (DateTime?)S(i));
collected.ShouldBe(reverse ? expected.Reverse() : expected);
}
/// <summary>The node manager's Raw paging loop at <c>NumValuesPerNode = 1</c> (see <c>ServeRawPagedAsync</c>).</summary>
private static async Task<List<DateTime?>> WalkAsync(IHistorianDataSource ds, DateTime start, DateTime end)
{
const uint cap = 1;
var res = ds.TimestampResolutionTicks;
var reverse = start > end;
var collected = new List<DateTime?>();
DateTime? cursor = start;
var skip = 0;
var resumed = false;
for (var pages = 0; cursor is { } from; pages++)
{
pages.ShouldBeLessThan(60, "paging must terminate, not loop");
var raw = (await ds.ReadRawAsync("P", from, end, cap, Ct)).Samples;
var full = raw.Count >= cap;
var page = raw.ToList();
if (resumed)
for (var dropped = 0; dropped < skip && page.Count > 0 && page[0].SourceTimestampUtc == from; dropped++)
page.RemoveAt(0);
if (resumed && full && (page.Count == 0 || page[^1].SourceTimestampUtc == from))
{
// Stalled on the boundary "cluster": point-read it, emit what is left, step off by the resolution.
var cluster = (await ds.ReadRawAsync("P", from, from, 1000, Ct)).Samples;
var sliceStart = Math.Min(skip, cluster.Count);
var slice = cluster.Skip(sliceStart).Take((int)cap).ToList();
collected.AddRange(slice.Select(s => s.SourceTimestampUtc));
if (sliceStart + slice.Count < cluster.Count) { skip = sliceStart + slice.Count; continue; }
var next = from.AddTicks(reverse ? -res : res);
cursor = reverse ? (next > end ? next : null) : (next <= end ? next : null);
skip = 0;
continue;
}
collected.AddRange(page.Select(s => s.SourceTimestampUtc));
if (!full || page.Count == 0) break;
cursor = page[^1].SourceTimestampUtc;
skip = page.Count(s => s.SourceTimestampUtc == cursor);
resumed = true;
}
return collected;
}
}
tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/ProvisioningTests.cs
+135
View File
@@ -0,0 +1,135 @@
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests;
[Collection(QuestDbCollection.Name)]
public sealed class ProvisioningTests(QuestDbFixture fx)
{
private (QuestDbTagProvisioner P, QuestDbConnection C, QuestDbTagCatalog Cat) Make(string? prefix = null, string? connectionString = null,
bool autoCreate = true)
{
var o = fx.Options(prefix ?? fx.UniquePrefix(), autoCreate: autoCreate);
if (connectionString is not null) o.ConnectionString = connectionString;
var c = new QuestDbConnection(o);
var b = new QuestDbSchemaBootstrapper(c, o);
var wal = new QuestDbWalVisibility(c, o.VisibilityTimeout);
var cat = new QuestDbTagCatalog(c, b, wal);
return (new QuestDbTagProvisioner(c, b, cat, wal), c, cat);
}
private static async Task<(byte Type, string? Eu, string? Desc)> TagAsync(QuestDbConnection c, string name)
{
await using var conn = await c.OpenAsync(default);
await using var cmd = c.CreateCommand(conn, $"SELECT data_type, eng_unit, description FROM {c.TagTable} WHERE name = $1");
QuestDbConnection.BindText(cmd, name);
await using var r = await cmd.ExecuteReaderAsync();
(await r.ReadAsync()).ShouldBeTrue();
return (QuestDbConnection.ReadKind(r, 0), r.IsDBNull(1) ? null : r.GetString(1), r.IsDBNull(2) ? null : r.GetString(2));
}
private static async Task<long> CountAsync(QuestDbConnection c)
{
await using var conn = await c.OpenAsync(default);
await using var cmd = c.CreateCommand(conn, $"SELECT count() FROM {c.TagTable}");
return Convert.ToInt64(await cmd.ExecuteScalarAsync());
}
[Fact]
public async Task Upsert_creates_then_changes_DataType_on_rerun_and_every_type_is_accepted()
{
fx.SkipIfUnavailable();
var (p, c, _) = Make();
var r1 = await p.EnsureTagsAsync([new("a", DriverDataType.Boolean, "degC", "desc"), new("s", DriverDataType.String, null, null)], CancellationToken.None);
r1.ShouldBe(new HistorianProvisionResult(2, 2, 0, 0));
(await TagAsync(c, "a")).ShouldBe(((byte)DriverDataType.Boolean, "degC", "desc"));
var r2 = await p.EnsureTagsAsync([new("a", DriverDataType.Float64, null, null)], CancellationToken.None);
r2.ShouldBe(new HistorianProvisionResult(1, 1, 0, 0));
(await TagAsync(c, "a")).ShouldBe(((byte)DriverDataType.Float64, "degC", "desc")); // null keeps stored metadata
(await CountAsync(c)).ShouldBe(2); // upsert by name, not a second row
}
[Fact]
public async Task A_request_with_values_replaces_only_the_fields_it_carries()
{
fx.SkipIfUnavailable();
var (p, c, _) = Make();
await p.EnsureTagsAsync([new("m", DriverDataType.Int32, "A", "first")], CancellationToken.None);
await p.EnsureTagsAsync([new("m", DriverDataType.Int32, "B", null)], CancellationToken.None);
(await TagAsync(c, "m")).ShouldBe(((byte)DriverDataType.Int32, "B", "first"));
}
[Fact]
public async Task Duplicate_names_and_bad_names_are_tallied_not_thrown()
{
fx.SkipIfUnavailable();
var (p, _, _) = Make();
var r = await p.EnsureTagsAsync([
new("d", DriverDataType.Int32, null, null), new("d", DriverDataType.Int64, null, null),
new("", DriverDataType.Int32, null, null), new(new string('x', 401), DriverDataType.Int32, null, null)], CancellationToken.None);
r.ShouldBe(new HistorianProvisionResult(4, 2, 0, 2));
}
[Fact]
public async Task More_tags_than_one_chunk_all_land()
{
fx.SkipIfUnavailable();
var (p, c, _) = Make();
var reqs = Enumerable.Range(0, 1000).Select(i => new HistorianTagProvisionRequest($"bulk{i}", DriverDataType.Float32, null, null)).ToList();
(await p.EnsureTagsAsync(reqs, CancellationToken.None)).ShouldBe(new HistorianProvisionResult(1000, 1000, 0, 0));
(await CountAsync(c)).ShouldBe(1000);
}
[Fact]
public async Task A_later_catalog_Ensure_of_a_provisioned_tag_does_not_wipe_its_metadata()
{
fx.SkipIfUnavailable();
var (p, c, cat) = Make();
await p.EnsureTagsAsync([new("k", DriverDataType.Float64, "kg", "mass")], CancellationToken.None);
(await cat.ExistsAsync("k", CancellationToken.None)).ShouldBeTrue();
await cat.EnsureAsync("k", 1, CancellationToken.None);
(await TagAsync(c, "k")).ShouldBe(((byte)DriverDataType.Float64, "kg", "mass"));
}
/// <summary>
/// Final review M-1: the catalog's first sighting of a tag can run while the provisioner's row is committed but
/// not yet applied (its WAL-visibility window). An unconditional catalog INSERT then lands after it and the
/// dedup key replaces the provisioner's row with one carrying no unit/description. The provisioner here does
/// not wait (VisibilityTimeout 0) and has its own catalog, so every catalog Ensure runs inside that window.
/// </summary>
[Fact]
public async Task A_catalog_Ensure_inside_the_provisioners_visibility_window_keeps_its_metadata()
{
fx.SkipIfUnavailable();
var o = fx.Options(fx.UniquePrefix());
await using var c = new QuestDbConnection(o);
var b = new QuestDbSchemaBootstrapper(c, o);
await b.EnsureAsync(CancellationToken.None);
var noWait = new QuestDbWalVisibility(c, TimeSpan.Zero);
var provisioner = new QuestDbTagProvisioner(c, b, new QuestDbTagCatalog(c, b, noWait), noWait);
var catalog = new QuestDbTagCatalog(c, b, new QuestDbWalVisibility(c, o.VisibilityTimeout));
const int Tags = 25;
for (var i = 0; i < Tags; i++)
{
var name = $"race{i}";
(await provisioner.EnsureTagsAsync([new(name, DriverDataType.Float64, "kg", $"mass {i}")], CancellationToken.None))
.Ensured.ShouldBe(1);
await catalog.EnsureAsync(name, 1, CancellationToken.None); // the recorder's first value for the tag
}
await new QuestDbWalVisibility(c, o.VisibilityTimeout).WaitAsync(c.TagTable, CancellationToken.None);
for (var i = 0; i < Tags; i++)
(await TagAsync(c, $"race{i}")).ShouldBe(((byte)DriverDataType.Float64, "kg", $"mass {i}"), $"race{i}");
(await CountAsync(c)).ShouldBe(Tags);
}
[Fact]
public async Task A_connection_failure_is_a_Failed_tally_not_an_exception()
{
fx.SkipIfUnavailable();
var (p, _, _) = Make(connectionString: "Host=localhost;Port=1;Username=admin;Password=quest;Database=qdb;ServerCompatibilityMode=NoTypeLoading;Timeout=2");
(await p.EnsureTagsAsync([new("x", DriverDataType.Int32, null, null)], CancellationToken.None)).ShouldBe(new HistorianProvisionResult(1, 0, 0, 1));
}
}
tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/QuestDbConformance.cs
+55
View File
@@ -0,0 +1,55 @@
using ZB.MOM.WW.OtOpcUa.Driver.Historian.Conformance;
using Xunit;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests;
// One concrete subclass per conformance class: the suite's tests run here against QuestDB.
[Collection(QuestDbCollection.Name)]
public sealed class QuestDbRawReadConformance(QuestDbFixture fx) : RawReadConformance
{
/// <inheritdoc/>
protected override Task<IConformanceBackend> CreateBackendAsync() => QuestDbConformanceBackend.CreateAsync(fx);
}
[Collection(QuestDbCollection.Name)]
public sealed class QuestDbAtTimeConformance(QuestDbFixture fx) : AtTimeConformance
{
/// <inheritdoc/>
protected override Task<IConformanceBackend> CreateBackendAsync() => QuestDbConformanceBackend.CreateAsync(fx);
}
[Collection(QuestDbCollection.Name)]
public sealed class QuestDbProcessedConformance(QuestDbFixture fx) : ProcessedConformance
{
/// <inheritdoc/>
protected override Task<IConformanceBackend> CreateBackendAsync() => QuestDbConformanceBackend.CreateAsync(fx);
}
[Collection(QuestDbCollection.Name)]
public sealed class QuestDbQualityContractConformance(QuestDbFixture fx) : QualityContractConformance
{
/// <inheritdoc/>
protected override Task<IConformanceBackend> CreateBackendAsync() => QuestDbConformanceBackend.CreateAsync(fx);
}
[Collection(QuestDbCollection.Name)]
public sealed class QuestDbLiveWriteConformance(QuestDbFixture fx) : LiveWriteConformance
{
/// <inheritdoc/>
protected override Task<IConformanceBackend> CreateBackendAsync() => QuestDbConformanceBackend.CreateAsync(fx);
}
[Collection(QuestDbCollection.Name)]
public sealed class QuestDbUpdateConformance(QuestDbFixture fx) : UpdateConformance
{
/// <inheritdoc/>
protected override Task<IConformanceBackend> CreateBackendAsync() => QuestDbConformanceBackend.CreateAsync(fx);
}
[Collection(QuestDbCollection.Name)]
public sealed class QuestDbTagNameConformance(QuestDbFixture fx) : TagNameConformance
{
/// <inheritdoc/>
protected override Task<IConformanceBackend> CreateBackendAsync() => QuestDbConformanceBackend.CreateAsync(fx);
}
tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/QuestDbConformanceBackend.cs
+202
View File
@@ -0,0 +1,202 @@
using System.Text;
using Npgsql;
using NpgsqlTypes;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.Driver.Historian.Conformance;
using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests;
/// <summary>
/// The QuestDB historian as a conformance backend: a fresh unique table prefix per instance (dropped on dispose),
/// seeded by plain multi-row INSERTs (never through the writer or updater) followed by the WAL visibility wait.
/// Storage resolution is 1 µs (10 ticks); <see cref="Round"/> is <see cref="QuestDbConnection.Round"/>, and
/// <see cref="SeedAsync"/> rounds the times it is given the same way before storing them.
/// </summary>
internal sealed class QuestDbConformanceBackend : IConformanceBackend
{
private const int SeedBatchRows = 1000;
private readonly QuestDbFixture _fx;
private readonly string _prefix;
private QuestDbConformanceBackend(QuestDbFixture fx, string prefix, QuestDbHistorianOptions options,
QuestDbConnection conn, QuestDbSchemaBootstrapper boot)
{
_fx = fx;
_prefix = prefix;
Options = options;
Conn = conn;
Boot = boot;
Wal = new QuestDbWalVisibility(conn, options.VisibilityTimeout);
}
/// <summary>The options of this instance's table set.</summary>
public QuestDbHistorianOptions Options { get; }
/// <summary>The connection helper.</summary>
public QuestDbConnection Conn { get; }
/// <summary>The schema bootstrapper (already run).</summary>
public QuestDbSchemaBootstrapper Boot { get; }
/// <summary>The WAL visibility wait.</summary>
public QuestDbWalVisibility Wal { get; }
/// <summary>Creates a backend over a fresh table prefix; skips the calling test when the gate is closed.</summary>
/// <param name="fx">The shared fixture.</param>
/// <returns>The backend.</returns>
public static async Task<IConformanceBackend> CreateAsync(QuestDbFixture fx) => await CreateConcreteAsync(fx);
/// <summary>As <see cref="CreateAsync"/>, typed (for QuestDB-specific tests that need the connection).</summary>
/// <param name="fx">The shared fixture.</param>
/// <returns>The backend.</returns>
public static async Task<QuestDbConformanceBackend> CreateConcreteAsync(QuestDbFixture fx)
{
fx.SkipIfUnavailable();
var prefix = fx.UniquePrefix();
var o = fx.Options(prefix);
var conn = new QuestDbConnection(o);
var boot = new QuestDbSchemaBootstrapper(conn, o);
await boot.EnsureAsync(CancellationToken.None);
return new QuestDbConformanceBackend(fx, prefix, o, conn, boot);
}
/// <inheritdoc/>
public long ResolutionTicks => 10;
/// <inheritdoc/>
public DateTime Round(DateTime t) => QuestDbConnection.Round(t);
/// <summary>A fresh tag catalog (fresh positive cache) over this table set.</summary>
public QuestDbTagCatalog NewCatalog() => new(Conn, Boot, Wal);
/// <inheritdoc/>
public IHistorianDataSource CreateDataSource(TimeProvider? clock = null) =>
new QuestDbHistorianDataSource(Conn, Boot, NewCatalog(), timeProvider: clock);
/// <inheritdoc/>
/// <remarks>The T4 live-value writer over a fresh catalog (fresh positive cache) and this instance's WAL wait.</remarks>
public IHistorianValueWriter CreateValueWriter(Func<DateTime>? utcNow = null) =>
new QuestDbHistorianValueWriter(Conn, NewCatalog(), Wal, utcNow: utcNow);
/// <inheritdoc/>
/// <remarks>The T6 soft-delete updater over a fresh catalog and this instance's WAL wait.</remarks>
public IHistorianDataUpdater CreateUpdater() => new QuestDbHistorianDataUpdater(Conn, NewCatalog(), Wal);
/// <inheritdoc/>
/// <remarks>
/// Times are rounded to the microsecond (<see cref="Round"/>) before they are stored; the tag row is registered
/// through the catalog (data type 0, as the SQL Server seed does); every row is live (<c>deleted = false</c>);
/// <c>server_ts</c> = rounded time + 1 s. Waits for WAL visibility before returning.
/// </remarks>
public async Task SeedAsync(string tag, params (DateTime T, HistorianValue V, uint Status)[] samples)
{
await NewCatalog().EnsureAsync(tag, 0, CancellationToken.None);
await InsertRowsAsync(tag, samples.Select(s => (Round(s.T), s.V, s.Status, Deleted: false)).ToArray());
}
/// <summary>
/// Inserts raw rows (the tag must already be registered when reads should find it), including tombstones, then
/// waits for WAL visibility. A tombstone re-inserted at an existing <c>(ts, tag)</c> replaces that row (DEDUP).
/// </summary>
/// <param name="tag">The tag.</param>
/// <param name="rows">Rows: time (already rounded by the caller or not — it is bound through BindTimestamp), value, status, deleted.</param>
public async Task InsertRowsAsync(string tag, (DateTime T, HistorianValue V, uint Status, bool Deleted)[] rows)
{
if (rows.Length == 0) return;
await using (var c = await Conn.OpenAsync(CancellationToken.None))
{
for (var offset = 0; offset < rows.Length; offset += SeedBatchRows)
{
var batch = rows.Skip(offset).Take(SeedBatchRows).ToArray();
var sql = new StringBuilder(
$"INSERT INTO {Conn.SampleTable} (ts, tag, kind, num_value, int_value, str_value, status, deleted, server_ts) VALUES ");
await using var cmd = Conn.CreateCommand(c, "");
var n = 0;
for (var i = 0; i < batch.Length; i++)
{
var (t, v, status, deleted) = batch[i];
var col = SampleColumnMapper.ToColumns(v);
if (i > 0) sql.Append(", ");
sql.Append('(');
for (var j = 0; j < 9; j++) sql.Append(j == 0 ? "" : ", ").Append('$').Append(++n);
sql.Append(')');
QuestDbConnection.BindTimestamp(cmd, t);
QuestDbConnection.BindText(cmd, tag);
QuestDbConnection.BindByte(cmd, col.ValueKind);
cmd.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Double, Value = (object?)col.NumValue ?? DBNull.Value });
cmd.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Bigint, Value = (object?)col.IntValue ?? DBNull.Value });
QuestDbConnection.BindText(cmd, col.StrValue);
QuestDbConnection.BindLong(cmd, SampleColumnMapper.ToLongStatus(status));
cmd.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Boolean, Value = deleted });
QuestDbConnection.BindTimestamp(cmd, Round(t).AddSeconds(1));
}
cmd.CommandText = sql.ToString();
await cmd.ExecuteNonQueryAsync();
}
}
await Wal.WaitAsync(Conn.SampleTable, CancellationToken.None);
}
/// <summary>Runs a statement against this table set (test-only escape hatch for hand-made rows), then waits for WAL visibility of the sample table.</summary>
/// <param name="sql">The statement; parameters are bound positionally by <paramref name="bind"/>.</param>
/// <param name="bind">Binds the parameters.</param>
public async Task ExecuteAsync(string sql, Action<NpgsqlCommand> bind)
{
await using (var c = await Conn.OpenAsync(CancellationToken.None))
{
await using var cmd = Conn.CreateCommand(c, sql);
bind(cmd);
await cmd.ExecuteNonQueryAsync();
}
await Wal.WaitAsync(Conn.SampleTable, CancellationToken.None);
}
/// <inheritdoc/>
public async Task<IReadOnlyList<(DateTime T, HistorianValue V, uint Status)>> ReadBackAsync(string tag)
{
await using var c = await Conn.OpenAsync(CancellationToken.None);
await using var cmd = Conn.CreateCommand(c,
$"SELECT ts, kind, num_value, int_value, str_value, status FROM {Conn.SampleTable} " +
"WHERE tag = $1 AND NOT deleted ORDER BY ts");
QuestDbConnection.BindText(cmd, tag);
var list = new List<(DateTime, HistorianValue, uint)>();
await using var r = await cmd.ExecuteReaderAsync();
while (await r.ReadAsync())
{
var v = QuestDbHistorianDataSource.ToValue(QuestDbConnection.ReadKind(r, 1),
r.IsDBNull(2) ? null : r.GetDouble(2), r.IsDBNull(3) ? null : r.GetInt64(3), r.IsDBNull(4) ? null : r.GetString(4));
// Verbatim: no read-side clamp. A NULL status cannot be written by any code path; it reads as the raw 0 bits.
var status = r.IsDBNull(5) ? 0u : SampleColumnMapper.FromLongStatus(r.GetInt64(5));
list.Add((QuestDbConnection.ReadUtc(r, 0), v, status));
}
return list;
}
/// <inheritdoc/>
public async Task<bool> TagExistsAsync(string tag)
{
await using var c = await Conn.OpenAsync(CancellationToken.None);
await using var cmd = Conn.CreateCommand(c, $"SELECT count() FROM {Conn.TagTable} WHERE name = $1");
QuestDbConnection.BindText(cmd, tag);
return Convert.ToInt64(await cmd.ExecuteScalarAsync()) > 0;
}
/// <inheritdoc/>
public async Task<DriverDataType?> TagDataTypeAsync(string tag)
{
await using var c = await Conn.OpenAsync(CancellationToken.None);
await using var cmd = Conn.CreateCommand(c, $"SELECT data_type FROM {Conn.TagTable} WHERE name = $1 LIMIT 1");
QuestDbConnection.BindText(cmd, tag);
await using var r = await cmd.ExecuteReaderAsync();
return await r.ReadAsync() ? (DriverDataType)QuestDbConnection.ReadKind(r, 0) : null;
}
/// <inheritdoc/>
public async ValueTask DisposeAsync()
{
await Conn.DisposeAsync();
await _fx.DropPrefixAsync(_prefix); // early; the fixture also drops every prefix it handed out
}
}
tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/QuestDbFixture.cs
+116
View File
@@ -0,0 +1,116 @@
using System.Collections.Concurrent;
using Npgsql;
using Xunit;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests;
/// <summary>
/// Env-gated QuestDB for the historian integration tests. <c>HISTORIAN_QUESTDB_TEST_CONNECTION</c> absent ⇒
/// <see cref="SkipReason"/> is set, nothing connects and every test calls <c>Assert.Skip</c>. Present ⇒ tests isolate
/// from each other with <see cref="UniquePrefix"/>; every prefix handed out is dropped on dispose (and a test may
/// drop its own early with <see cref="DropPrefixAsync"/>). Never touches a table outside those prefixes.
/// </summary>
public sealed class QuestDbFixture : IAsyncLifetime
{
/// <summary>Env var holding an Npgsql connection string for QuestDB's PG-wire port.</summary>
public const string ConnectionEnvVar = "HISTORIAN_QUESTDB_TEST_CONNECTION";
private readonly ConcurrentBag<string> _prefixes = new();
/// <summary>Why tests must skip, or null when QuestDB is available.</summary>
public string? SkipReason { get; private set; }
/// <summary>The Npgsql connection string. Valid only when <see cref="SkipReason"/> is null.</summary>
public string ConnectionString { get; private set; } = "";
/// <summary>Skips the calling test when the gate is closed.</summary>
public void SkipIfUnavailable()
{
if (SkipReason is not null) Assert.Skip(SkipReason);
}
/// <summary>A fresh, valid table prefix (<c>"t" + 12 hex + "_"</c>) that nothing uses yet; dropped on dispose.</summary>
/// <returns>The prefix.</returns>
public string UniquePrefix()
{
var p = "t" + Guid.NewGuid().ToString("N")[..12] + "_";
_prefixes.Add(p);
return p;
}
/// <summary>Options targeting this fixture's QuestDB under <paramref name="prefix"/>.</summary>
/// <param name="prefix">Table prefix (normally from <see cref="UniquePrefix"/>).</param>
/// <param name="retentionDays">Optional TTL.</param>
/// <param name="autoCreate">AutoCreateSchema.</param>
/// <returns>Valid options.</returns>
/// <remarks>
/// Pooled connections idle for more than a second are closed (<c>Connection Idle Lifetime</c> /
/// <c>Connection Pruning Interval</c>). Many tests build their own <see cref="QuestDbConnection"/> (one Npgsql
/// data source each) and never dispose it; with Npgsql's default 300 s idle lifetime their pooled connections
/// piled up until QuestDB's PG-wire limit (64) was reached and later tests' opens timed out after 15 s.
/// </remarks>
public QuestDbHistorianOptions Options(string prefix, int? retentionDays = null, bool autoCreate = true) => new()
{
ConnectionString = new NpgsqlConnectionStringBuilder(ConnectionString)
{
ConnectionIdleLifetime = 1,
ConnectionPruningInterval = 1,
}.ConnectionString,
TablePrefix = prefix,
RetentionDays = retentionDays,
AutoCreateSchema = autoCreate,
};
/// <summary>Drops every table whose name starts with <paramref name="prefix"/> (found through <c>tables()</c>).</summary>
/// <param name="prefix">Table prefix.</param>
/// <returns>A task completing when they are gone.</returns>
public async Task DropPrefixAsync(string prefix)
{
if (SkipReason is not null) return;
if (!QuestDbHistorianOptions.IsValidPrefix(prefix)) throw new ArgumentException("Invalid prefix.", nameof(prefix));
await using var ds = NpgsqlDataSource.Create(ConnectionString);
await using var conn = await ds.OpenConnectionAsync().ConfigureAwait(false);
var names = new List<string>();
await using (var cmd = conn.CreateCommand())
{
cmd.CommandText = "SELECT table_name FROM tables() WHERE starts_with(table_name, $1)"; // never LIKE: '_' is a wildcard
cmd.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlTypes.NpgsqlDbType.Varchar, Value = prefix });
await using var r = await cmd.ExecuteReaderAsync().ConfigureAwait(false);
while (await r.ReadAsync().ConfigureAwait(false)) names.Add(r.GetString(0));
}
foreach (var n in names)
{
await using var drop = conn.CreateCommand();
drop.CommandText = $"DROP TABLE IF EXISTS {n}";
await drop.ExecuteNonQueryAsync().ConfigureAwait(false);
}
}
/// <inheritdoc/>
public ValueTask InitializeAsync()
{
var cs = Environment.GetEnvironmentVariable(ConnectionEnvVar);
if (string.IsNullOrWhiteSpace(cs))
SkipReason = $"{ConnectionEnvVar} is not set; QuestDB historian integration tests skipped.";
else
ConnectionString = cs;
return ValueTask.CompletedTask;
}
/// <inheritdoc/>
public async ValueTask DisposeAsync()
{
if (SkipReason is not null) return;
foreach (var p in _prefixes.Distinct())
await DropPrefixAsync(p).ConfigureAwait(false);
NpgsqlConnection.ClearAllPools();
}
}
/// <summary>Shares one fixture across the QuestDB integration tests.</summary>
[CollectionDefinition(Name)]
public sealed class QuestDbCollection : ICollectionFixture<QuestDbFixture>
{
/// <summary>Collection name.</summary>
public const string Name = "QuestDb";
}
tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/ReadTests.cs
+266
View File
@@ -0,0 +1,266 @@
using System.Text;
using Npgsql;
using NpgsqlTypes;
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.Driver.Historian.Conformance;
using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests;
/// <summary>
/// QuestDB-specific read behaviour (the shared Raw/AtTime/Processed literals run through the conformance suite):
/// tombstones, the pre-1970 / whole-range bucket math, reverse paging at one value per page, QuestDB's LONG NULL
/// sentinel, the kind-decides-the-value rule, AtTime chunking, Events and failure accounting.
/// </summary>
[Collection(QuestDbCollection.Name)]
public sealed class ReadTests(QuestDbFixture fx) : IAsyncLifetime
{
private static readonly CancellationToken Ct = CancellationToken.None;
private const uint Good = ConformanceData.Good;
private const uint Bad = ConformanceData.Bad;
private const uint GoodCalc = 0x00000401u;
private const uint UncertainCalc = 0x40A40401u;
private const uint UncertainInterpolated = 0x40A40402u;
private const uint GoodInterpolated = 0x00000402u;
private const uint NoDataCalc = 0x809B0401u;
private QuestDbConformanceBackend _b = null!;
private static DateTime At(double s) => ConformanceData.At(s);
public async ValueTask InitializeAsync() => _b = await QuestDbConformanceBackend.CreateConcreteAsync(fx);
public async ValueTask DisposeAsync()
{
if (_b is not null) await _b.DisposeAsync();
}
private static double?[] Values(HistoryReadResult r) => r.Samples.Select(s => (double?)s.Value).ToArray();
/// <summary>What T6's DeleteAtTime writes: the same (ts, tag), deleted = true, value columns NULL, status BadNoData.</summary>
private Task TombstoneAsync(string tag, DateTime t) =>
_b.InsertRowsAsync(tag, [(t, HistorianValue.Empty, HistorianStatusCodes.BadNoData, true)]);
[Fact]
public async Task Tombstoned_rows_are_invisible_to_raw_attime_and_processed()
{
await ConformanceData.SeedAAsync(_b); // +0=10, +10=20, +20=Bad 30, +30=40, +40=50
await TombstoneAsync("A", At(10)); // DEDUP replaces the +10 row with a tombstone
(await _b.ReadBackAsync("A")).Count.ShouldBe(4);
var src = _b.CreateDataSource();
var raw = await src.ReadRawAsync("A", At(0), At(100), 0, Ct);
Values(raw).ShouldBe([10d, 30d, 40d, 50d]);
raw.Samples.Select(s => s.StatusCode).ShouldBe([Good, Bad, Good, Good]);
// +10 is no longer an exact hit: previous is +0 = 10 (Good), next is the Bad +20 ⇒ held, Uncertain.
var at = (await src.ReadAtTimeAsync("A", [At(10), At(5)], Ct)).Samples;
at.Select(s => s.Value).ShouldBe([(object?)10d, 10d]);
at.ShouldAllBe(s => s.StatusCode == UncertainInterpolated);
// B0 [0,20) holds only {10}: Good, count 1. A visible tombstone would be a Bad sample ⇒ Uncertain, PercentBad 50.
async Task<DataValueSnapshot> B0(HistoryAggregateType a) =>
(await src.ReadProcessedAsync("A", At(0), At(80), TimeSpan.FromSeconds(20), a, Ct)).Samples[0];
var avg = await B0(HistoryAggregateType.Average);
avg.Value.ShouldBe(10d);
avg.StatusCode.ShouldBe(GoodCalc);
(await B0(HistoryAggregateType.Count)).Value.ShouldBe(1);
(await B0(HistoryAggregateType.PercentBad)).Value.ShouldBe(0d);
(await B0(HistoryAggregateType.End)).Value.ShouldBe(10d);
}
[Fact]
public async Task A_whole_range_processed_read_from_1601_to_9999_with_interval_zero_does_not_overflow()
{
await ConformanceData.SeedAAsync(_b);
var src = _b.CreateDataSource();
var start = new DateTime(1601, 1, 1, 0, 0, 0, DateTimeKind.Utc);
var end = new DateTime(9999, 12, 31, 0, 0, 0, DateTimeKind.Utc);
foreach (var agg in new[] { HistoryAggregateType.Average, HistoryAggregateType.Start, HistoryAggregateType.MaximumActualTime })
{
var s = (await src.ReadProcessedAsync("A", start, end, TimeSpan.Zero, agg, Ct)).Samples.Single();
s.StatusCode.ShouldBe(UncertainCalc, agg.ToString());
}
(await src.ReadProcessedAsync("A", start, end, TimeSpan.Zero, HistoryAggregateType.Average, Ct)).Samples.Single().Value.ShouldBe(30d);
(await src.ReadProcessedAsync("A", start, end, TimeSpan.Zero, HistoryAggregateType.Start, Ct)).Samples.Single().Value.ShouldBe(10d);
var max = (await src.ReadProcessedAsync("A", start, end, TimeSpan.Zero, HistoryAggregateType.MaximumActualTime, Ct)).Samples.Single();
max.Value.ShouldBe(50d);
max.SourceTimestampUtc.ShouldBe(_b.Round(At(40)));
// The extreme DateTime range: MinValue/MaxValue are nudged (never bound as ±infinity, which would match nothing).
(await src.ReadProcessedAsync("A", DateTime.MinValue, DateTime.MaxValue, TimeSpan.Zero, HistoryAggregateType.Total, Ct))
.Samples.Single().Value.ShouldBe(120d);
// A pre-1970 window holds no data and is not an error.
(await src.ReadProcessedAsync("A", start, new DateTime(1969, 1, 1, 0, 0, 0, DateTimeKind.Utc), TimeSpan.FromDays(3650),
HistoryAggregateType.Count, Ct)).Samples.ShouldAllBe(x => x.StatusCode == NoDataCalc);
src.GetHealthSnapshot().TotalFailures.ShouldBe(0);
}
/// <summary>
/// The node manager's reverse paging at <c>NumValuesPerNode = 1</c> (OtOpcUaNodeManager + HistoryPaging): resume at
/// the last timestamp inclusive and trim it; when the trimmed page stalls on that timestamp, point-read it and step
/// one tick down. One tick below a µs-aligned time rounds to the previous µs, so the walk advances.
/// </summary>
[Fact]
public async Task A_reverse_paging_walk_with_one_value_per_page_returns_every_row()
{
await ConformanceData.SeedAAsync(_b);
var src = _b.CreateDataSource();
var end = At(-1);
DateTime start = At(100);
var resumed = false;
var emitted = new List<object?>();
for (var guard = 0; guard < 50; guard++)
{
var page = (await src.ReadRawAsync("A", start, end, 1, Ct)).Samples;
var trimmed = resumed ? page.Where(s => s.SourceTimestampUtc != start).ToList() : page.ToList();
if (page.Count == 0) break;
if (resumed && (trimmed.Count == 0 || trimmed[^1].SourceTimestampUtc == start))
{
// Stalled on the boundary: the tie cluster at `start` is one sample (already emitted) ⇒ step down a tick.
(await src.ReadRawAsync("A", start, start, 2, Ct)).Samples.Count.ShouldBe(1);
if (start.AddTicks(-1) <= end) break;
start = start.AddTicks(-1);
resumed = false;
continue;
}
emitted.AddRange(trimmed.Select(s => s.Value));
start = page[^1].SourceTimestampUtc!.Value;
resumed = true;
}
emitted.ShouldBe([50d, 40d, 30d, 20d, 10d]);
}
[Fact]
public async Task An_int64_of_long_MinValue_is_served_as_long_MinValue_not_zero()
{
// QuestDB stores long.MinValue in a LONG as NULL (its NULL sentinel); num_value keeps -2^63 exactly (RC2).
await _b.SeedAsync("L",
(At(0), HistorianValue.FromInt64(long.MinValue), Good),
(At(1), HistorianValue.FromInt64(long.MinValue + 1), Good));
var src = _b.CreateDataSource();
(await src.ReadRawAsync("L", At(0), At(10), 0, Ct)).Samples.Select(s => s.Value)
.ShouldBe([(object?)long.MinValue, long.MinValue + 1]);
(await src.ReadAtTimeAsync("L", [At(0)], Ct)).Samples.Single().Value.ShouldBe(long.MinValue);
(await src.ReadProcessedAsync("L", At(0), At(10), TimeSpan.Zero, HistoryAggregateType.Start, Ct))
.Samples.Single().Value.ShouldBe(long.MinValue);
}
[Fact]
public async Task The_kind_not_the_populated_column_decides_the_value()
{
// An Int64-kind row whose num_value is NULL (a NaN-free double view lost) still serves its int_value, and a
// Double-kind row carrying a stray int_value still serves num_value.
await _b.SeedAsync("K");
await _b.ExecuteAsync(
$"INSERT INTO {_b.Conn.SampleTable} (ts, tag, kind, num_value, int_value, status, deleted, server_ts) " +
"VALUES ($1, $2, $3, NULL, $4, $5, false, $6), ($7, $2, $8, $9, $10, $5, false, $11)",
cmd =>
{
QuestDbConnection.BindTimestamp(cmd, At(0)); // $1
QuestDbConnection.BindText(cmd, "K"); // $2
QuestDbConnection.BindByte(cmd, (byte)HistorianValueKind.Int64); // $3
QuestDbConnection.BindLong(cmd, 7); // $4
QuestDbConnection.BindLong(cmd, 0); // $5 status Good
QuestDbConnection.BindTimestamp(cmd, At(1)); // $6
QuestDbConnection.BindTimestamp(cmd, At(2)); // $7
QuestDbConnection.BindByte(cmd, (byte)HistorianValueKind.Double); // $8
cmd.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Double, Value = 2.5 }); // $9
QuestDbConnection.BindLong(cmd, 99); // $10
QuestDbConnection.BindTimestamp(cmd, At(3)); // $11
});
var s = (await _b.CreateDataSource().ReadRawAsync("K", At(0), At(10), 0, Ct)).Samples;
s.Select(x => x.Value).ShouldBe([(object?)7L, 2.5d]);
}
[Fact]
public async Task AtTime_over_several_statement_chunks_keeps_request_length_and_order()
{
await ConformanceData.SeedAAsync(_b);
// 140 distinct times (3 statements of ≤ 50), requested in descending order, plus a duplicate of the first.
var times = Enumerable.Range(0, 140).Select(i => At((390 - i * 3) / 10.0)).ToList();
times.Add(times[0]);
var s = (await _b.CreateDataSource().ReadAtTimeAsync("A", times, Ct)).Samples;
s.Count.ShouldBe(times.Count);
s.Select(x => x.SourceTimestampUtc).ShouldBe(times.Select(t => (DateTime?)t)); // the requested times (M3)
s[0].Value.ShouldBe(49d); // +39: 40 + 10 × 9/10
s[0].StatusCode.ShouldBe(GoodInterpolated);
s[^1].Value.ShouldBe(49d); // the duplicate
s[100].Value.ShouldBe(19d); // +9: 10 + 10 × 9/10, from the third chunk
s[139].Value.ShouldBeNull(); // -2.7 s: no previous sample
s[139].StatusCode.ShouldBe(0x809B0000u); // BadNoData
}
[Fact]
public async Task Events_are_filtered_by_source_and_window_ascending_and_limited_with_a_continuation()
{
await _b.Boot.EnsureAsync(Ct);
async Task Event(DateTime t, string id, string source, int severity, string? message)
{
await _b.ExecuteAsync(
$"INSERT INTO {_b.Conn.EventTable} (ts, event_id, source, severity, message, received_utc) VALUES ($1, $2, $3, $4, $5, $6)",
cmd =>
{
QuestDbConnection.BindTimestamp(cmd, t);
QuestDbConnection.BindText(cmd, id);
QuestDbConnection.BindText(cmd, source);
cmd.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Integer, Value = severity });
QuestDbConnection.BindText(cmd, message);
QuestDbConnection.BindTimestamp(cmd, t.AddSeconds(2));
});
await _b.Wal.WaitAsync(_b.Conn.EventTable, Ct);
}
await Event(At(3), "e3", "src1", 700, "three");
await Event(At(1), "e1", "src1", 70000, "one");
await Event(At(2), "e2", "src2", 500, null);
await Event(At(9), "e9", "src1", 500, "out of window");
var src = _b.CreateDataSource();
var all = await src.ReadEventsAsync(null, At(0), At(5), 0, Ct);
all.Events.Select(e => e.EventId).ShouldBe(["e1", "e2", "e3"]);
all.ContinuationPoint.ShouldBeNull();
all.Events[0].Severity.ShouldBe((ushort)65535);
all.Events[0].EventTimeUtc.ShouldBe(_b.Round(At(1)));
all.Events[0].ReceivedTimeUtc.ShouldBe(_b.Round(At(1)).AddSeconds(2));
all.Events[1].Message.ShouldBeNull();
var one = await src.ReadEventsAsync("src1", At(5), At(0), 1, Ct); // reversed window is swapped
one.Events.Select(e => e.EventId).ShouldBe(["e1"]);
Encoding.UTF8.GetString(one.ContinuationPoint!).ShouldBe(EventContinuation.Encode(_b.Round(At(3)), "e3"));
(await src.ReadEventsAsync("src1", At(0), At(5), 2, Ct)).ContinuationPoint.ShouldBeNull();
(await src.ReadEventsAsync(new string('x', 401), At(0), At(5), 0, Ct)).Events.ShouldBeEmpty();
}
[Fact]
public async Task A_failed_read_is_rethrown_and_recorded_in_health_without_counting_client_errors()
{
// A table set that does not exist, with AutoCreateSchema off: every read that reaches the database fails.
var prefix = fx.UniquePrefix();
var o = fx.Options(prefix, autoCreate: false);
await using var conn = new QuestDbConnection(o);
var boot = new QuestDbSchemaBootstrapper(conn, o);
using var src = new QuestDbHistorianDataSource(conn, boot, new QuestDbTagCatalog(conn, boot, new QuestDbWalVisibility(conn, o.VisibilityTimeout)));
await Should.ThrowAsync<HistorianSchemaMissingException>(() => src.ReadRawAsync("A", At(0), At(100), 0, Ct));
await Should.ThrowAsync<HistorianSchemaMissingException>(() => src.ReadEventsAsync(null, At(0), At(5), 0, Ct));
// Client-request errors are thrown before any I/O and are not backend failures.
await Should.ThrowAsync<HistoryAggregateNotSupportedException>(() =>
src.ReadProcessedAsync("A", At(0), At(5), TimeSpan.Zero, HistoryAggregateType.Interpolative, Ct));
await Should.ThrowAsync<ArgumentOutOfRangeException>(() =>
src.ReadProcessedAsync("A", At(0), At(100), TimeSpan.FromTicks(10), HistoryAggregateType.Count, Ct));
var h = src.GetHealthSnapshot();
h.TotalQueries.ShouldBe(2);
h.TotalFailures.ShouldBe(2);
h.ConsecutiveFailures.ShouldBe(2);
h.LastError.ShouldNotBeNull();
h.LastError.ShouldStartWith(nameof(HistorianSchemaMissingException));
h.ProcessConnectionOpen.ShouldBeFalse();
// The healthy backend's own data source is unaffected (a failure fails only its call).
await ConformanceData.SeedAAsync(_b);
(await _b.CreateDataSource().ReadRawAsync("A", At(0), At(100), 0, Ct)).Samples.Count.ShouldBe(5);
}
}
tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/TombstoneTests.Redelivery.cs
+38
View File
@@ -0,0 +1,38 @@
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests;
public sealed partial class TombstoneTests
{
/// <summary>
/// The live writer's outbox redelivers batches (at-least-once). A batch replayed after an operator's
/// DeleteAtTime must not bring the deleted sample back: the writer's existence check counts a tombstone as
/// present (it deliberately does not filter <c>NOT deleted</c>), so the redelivered row is skipped.
/// </summary>
[Fact]
public async Task Tombstone_is_not_resurrected_by_a_writer_redelivery()
{
fx.SkipIfUnavailable();
var rig = UpdateTests.MakeRig(fx);
var writer = new QuestDbHistorianValueWriter(rig.Conn, rig.Tags,
new QuestDbWalVisibility(rig.Conn, TimeSpan.FromSeconds(5)));
HistorizationValue[] batch =
[
new(T0, HistorianValue.FromDouble(1), HistorianStatusCodes.Good),
new(T0.AddSeconds(1), HistorianValue.FromDouble(2), HistorianStatusCodes.Good),
];
(await writer.WriteLiveValuesAsync("r", batch, default)).ShouldBeTrue();
(await rig.Updater.DeleteAtTimeAsync("r", [T0], default)).ShouldBe([HistorianStatusCodes.Good]);
(await writer.WriteLiveValuesAsync("r", batch, default)).ShouldBeTrue(); // the redelivery
(await UpdateTests.RowsAsync(rig, "r", liveOnly: true)).Select(r => r.Num).ShouldBe([2d]);
var all = await UpdateTests.RowsAsync(rig, "r");
all.Count.ShouldBe(2);
all[0].Deleted.ShouldBeTrue();
all[0].Num.ShouldBeNull();
}
}
tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/TombstoneTests.cs
+50
View File
@@ -0,0 +1,50 @@
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests;
/// <summary>Soft-delete behaviour. The writer-redelivery case lives in T7 (<c>TombstoneTests.Redelivery.cs</c>).</summary>
[Collection(QuestDbCollection.Name)]
public sealed partial class TombstoneTests(QuestDbFixture fx)
{
private static readonly DateTime T0 = new(2026, 3, 1, 12, 0, 0, DateTimeKind.Utc);
[Fact]
public async Task Delete_then_raw_read_hides_the_row_and_leaves_a_NULL_valued_tombstone()
{
fx.SkipIfUnavailable();
var rig = UpdateTests.MakeRig(fx);
await rig.Updater.UpdateDataAsync("t", HistoryUpdateMode.Insert,
[UpdateTests.V(T0, 1), UpdateTests.V(T0.AddSeconds(1), 2)], default);
(await rig.Updater.DeleteAtTimeAsync("t", [T0], default)).ShouldBe([HistorianStatusCodes.Good]);
(await UpdateTests.RowsAsync(rig, "t", liveOnly: true)).Select(r => r.Num).ShouldBe([2d]);
var all = await UpdateTests.RowsAsync(rig, "t");
all.Count.ShouldBe(2); // the tombstone replaced the row in place (dedup upsert)
var tomb = all[0];
tomb.Deleted.ShouldBeTrue();
tomb.Num.ShouldBeNull();
tomb.Kind.ShouldBe((byte)0);
tomb.Status.ShouldBe((long)HistorianStatusCodes.BadNoData);
}
[Fact]
public async Task Insert_over_a_tombstone_reports_GoodEntryInserted_and_the_row_is_visible()
{
fx.SkipIfUnavailable();
var rig = UpdateTests.MakeRig(fx);
await rig.Updater.UpdateDataAsync("t", HistoryUpdateMode.Insert, [UpdateTests.V(T0, 1)], default);
await rig.Updater.DeleteAtTimeAsync("t", [T0], default);
(await rig.Updater.UpdateDataAsync("t", HistoryUpdateMode.Replace, [UpdateTests.V(T0, 5)], default))
.ShouldBe([HistorianStatusCodes.BadNoEntryExists]); // a tombstone is no entry for Replace
(await rig.Updater.UpdateDataAsync("t", HistoryUpdateMode.Insert, [UpdateTests.V(T0, 9)], default))
.ShouldBe([HistorianStatusCodes.GoodEntryInserted]);
var live = await UpdateTests.RowsAsync(rig, "t", liveOnly: true);
live.Count.ShouldBe(1);
live[0].Num.ShouldBe(9d);
live[0].Deleted.ShouldBeFalse();
}
}
tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/UpdateTests.cs
+268
View File
@@ -0,0 +1,268 @@
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests;
/// <summary>QuestDB-specific checks of <see cref="QuestDbHistorianDataUpdater"/> (the literal codes run through conformance in T7).</summary>
[Collection(QuestDbCollection.Name)]
public sealed class UpdateTests(QuestDbFixture fx)
{
private static readonly DateTime T0 = new(2026, 3, 1, 12, 0, 0, DateTimeKind.Utc); // µs-aligned
private const uint Good = HistorianStatusCodes.Good;
internal sealed record Rig(QuestDbConnection Conn, QuestDbHistorianDataUpdater Updater, QuestDbTagCatalog Tags);
internal static Rig MakeRig(QuestDbFixture fx, TimeSpan? visibility = null)
{
var o = fx.Options(fx.UniquePrefix());
o.VisibilityTimeout = visibility ?? o.VisibilityTimeout;
var c = new QuestDbConnection(o);
var wal = new QuestDbWalVisibility(c, o.VisibilityTimeout);
var tags = new QuestDbTagCatalog(c, new QuestDbSchemaBootstrapper(c, o), wal);
return new Rig(c, new QuestDbHistorianDataUpdater(c, tags, wal), tags);
}
internal static HistorizationValue V(DateTime? t, double v, uint status = Good) =>
new(t, HistorianValue.FromDouble(v), status);
internal sealed record Row(DateTime Ts, byte Kind, double? Num, long? Status, bool Deleted);
/// <summary>Every stored row for the tag (tombstones included), straight from the table.</summary>
internal static async Task<List<Row>> RowsAsync(Rig rig, string tag, bool liveOnly = false)
{
await using var conn = await rig.Conn.OpenAsync(default);
await using var cmd = rig.Conn.CreateCommand(conn,
$"SELECT ts, kind, num_value, status, deleted FROM {rig.Conn.SampleTable} WHERE tag = $1 {(liveOnly ? "AND NOT deleted" : "")} ORDER BY ts");
QuestDbConnection.BindText(cmd, tag);
var rows = new List<Row>();
await using var r = await cmd.ExecuteReaderAsync();
while (await r.ReadAsync())
rows.Add(new Row(QuestDbConnection.ReadUtc(r, 0), QuestDbConnection.ReadKind(r, 1),
r.IsDBNull(2) ? null : r.GetDouble(2), r.IsDBNull(3) ? null : r.GetInt64(3), r.GetBoolean(4)));
return rows;
}
internal static async Task<long> TagRowsAsync(Rig rig, string tag)
{
await using var conn = await rig.Conn.OpenAsync(default);
await using var cmd = rig.Conn.CreateCommand(conn, $"SELECT count() FROM {rig.Conn.TagTable} WHERE name = $1");
QuestDbConnection.BindText(cmd, tag);
return Convert.ToInt64(await cmd.ExecuteScalarAsync());
}
[Fact]
public async Task Insert_writes_absent_and_refuses_existing_and_is_visible_immediately()
{
fx.SkipIfUnavailable();
var rig = MakeRig(fx);
(await rig.Updater.UpdateDataAsync("t1", HistoryUpdateMode.Insert, [V(T0, 1)], default))
.ShouldBe([HistorianStatusCodes.GoodEntryInserted]);
var codes = await rig.Updater.UpdateDataAsync("t1", HistoryUpdateMode.Insert,
[V(T0, 99), V(T0.AddSeconds(1), 2)], default);
codes.ShouldBe([HistorianStatusCodes.BadEntryExists, HistorianStatusCodes.GoodEntryInserted]);
var rows = await RowsAsync(rig, "t1"); // read right after return: read-your-writes
rows.Select(r => r.Num).ShouldBe([1d, 2d]);
(await TagRowsAsync(rig, "t1")).ShouldBe(1);
}
[Fact]
public async Task Replace_overwrites_present_refuses_missing_and_never_creates_the_tag()
{
fx.SkipIfUnavailable();
var rig = MakeRig(fx);
(await rig.Updater.UpdateDataAsync("nope", HistoryUpdateMode.Replace, [V(T0, 1)], default))
.ShouldBe([HistorianStatusCodes.BadNoEntryExists]);
(await TagRowsAsync(rig, "nope")).ShouldBe(0);
(await RowsAsync(rig, "nope")).ShouldBeEmpty();
await rig.Updater.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T0, 1)], default);
var codes = await rig.Updater.UpdateDataAsync("t", HistoryUpdateMode.Replace,
[V(T0, 5), V(T0.AddSeconds(1), 6)], default);
codes.ShouldBe([HistorianStatusCodes.GoodEntryReplaced, HistorianStatusCodes.BadNoEntryExists]);
(await RowsAsync(rig, "t")).Select(r => r.Num).ShouldBe([5d]);
}
[Fact]
public async Task Update_upserts_both_ways_and_last_duplicate_wins()
{
fx.SkipIfUnavailable();
var rig = MakeRig(fx);
await rig.Updater.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T0, 1)], default);
var codes = await rig.Updater.UpdateDataAsync("t", HistoryUpdateMode.Update,
[V(T0, 2), V(T0.AddSeconds(1), 3), V(T0.AddSeconds(1).AddTicks(3), 4)], default);
codes.ShouldBe([HistorianStatusCodes.GoodEntryReplaced, HistorianStatusCodes.GoodEntryInserted,
HistorianStatusCodes.GoodEntryInserted]);
(await RowsAsync(rig, "t")).Select(r => r.Num).ShouldBe([2d, 4d]);
}
[Fact]
public async Task Same_microsecond_duplicates_collapse_before_the_database_sees_them()
{
fx.SkipIfUnavailable();
var rig = MakeRig(fx);
// offsets 0 and 3 ticks sit inside one 10-tick window => the same stored µs
var codes = await rig.Updater.UpdateDataAsync("t", HistoryUpdateMode.Insert,
[V(T0, 1), V(T0.AddTicks(3), 2)], default);
codes.ShouldBe([HistorianStatusCodes.GoodEntryInserted, HistorianStatusCodes.BadEntryExists]);
(await RowsAsync(rig, "t")).Select(r => r.Num).ShouldBe([1d]);
}
[Fact]
public async Task Invalid_items_get_the_planners_codes_and_the_rest_proceed()
{
fx.SkipIfUnavailable();
var rig = MakeRig(fx);
var codes = await rig.Updater.UpdateDataAsync("t", HistoryUpdateMode.Insert,
[
V(null, 1), V(DateTime.MinValue, 1), V(T0, double.NaN),
new HistorizationValue(T0.AddSeconds(1), HistorianValue.Empty, Good),
V(T0.AddSeconds(2), 7),
], default);
codes.ShouldBe([HistorianStatusCodes.BadInvalidTimestamp, HistorianStatusCodes.BadInvalidTimestamp,
HistorianStatusCodes.BadInvalidArgument, HistorianStatusCodes.BadInvalidArgument,
HistorianStatusCodes.GoodEntryInserted]);
(await RowsAsync(rig, "t")).Count.ShouldBe(1);
}
[Fact]
public async Task Over_long_tag_is_refused_and_nothing_is_written()
{
fx.SkipIfUnavailable();
var rig = MakeRig(fx);
var tag = new string('x', HistorianTagNames.MaxLength + 1);
(await rig.Updater.UpdateDataAsync(tag, HistoryUpdateMode.Insert, [V(T0, 1)], default))
.ShouldBe([HistorianStatusCodes.BadInvalidArgument]);
(await rig.Updater.DeleteAtTimeAsync(tag, [T0], default)).ShouldBe([HistorianStatusCodes.BadInvalidArgument]);
(await rig.Updater.DeleteRawAsync(tag, T0, T0.AddHours(1), default)).ShouldBe(HistorianStatusCodes.BadInvalidArgument);
}
[Fact]
public async Task Pre_1970_items_are_BadOutOfRange_and_nothing_is_sent()
{
fx.SkipIfUnavailable();
var rig = MakeRig(fx);
var pre = new DateTime(1969, 12, 31, 23, 59, 59, DateTimeKind.Utc);
foreach (var mode in new[] { HistoryUpdateMode.Insert, HistoryUpdateMode.Replace, HistoryUpdateMode.Update })
{
var codes = await rig.Updater.UpdateDataAsync("t", mode, [V(pre, 1), V(DateTime.MinValue, 1), V(T0, 2)], default);
codes[0].ShouldBe(HistorianStatusCodes.BadOutOfRange);
codes[1].ShouldBe(HistorianStatusCodes.BadInvalidTimestamp);
}
// Pre-1970 only: the whole batch is refused, no tag row is created, no INSERT would succeed anyway.
(await rig.Updater.UpdateDataAsync("only-pre", HistoryUpdateMode.Update, [V(pre, 1)], default))
.ShouldBe([HistorianStatusCodes.BadOutOfRange]);
(await TagRowsAsync(rig, "only-pre")).ShouldBe(0);
(await rig.Updater.DeleteAtTimeAsync("t", [pre], default)).ShouldBe([HistorianStatusCodes.BadNoEntryExists]);
}
[Fact]
public async Task DeleteAtTime_tombstones_live_rows_and_reports_per_time_codes()
{
fx.SkipIfUnavailable();
var rig = MakeRig(fx);
await rig.Updater.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T0, 1), V(T0.AddSeconds(1), 2)], default);
var codes = await rig.Updater.DeleteAtTimeAsync("t",
[T0, T0.AddSeconds(5), T0, DateTime.MinValue], default);
codes.ShouldBe([Good, HistorianStatusCodes.BadNoEntryExists, HistorianStatusCodes.BadNoEntryExists,
HistorianStatusCodes.BadInvalidTimestamp]);
(await RowsAsync(rig, "t", liveOnly: true)).Select(r => r.Num).ShouldBe([2d]);
// a tombstone is not a live row: deleting it again finds nothing
(await rig.Updater.DeleteAtTimeAsync("t", [T0], default)).ShouldBe([HistorianStatusCodes.BadNoEntryExists]);
(await rig.Updater.DeleteAtTimeAsync("unknown", [T0], default)).ShouldBe([HistorianStatusCodes.BadNoEntryExists]);
}
[Fact]
public async Task DeleteAtTime_with_a_sub_microsecond_request_time_still_reports_Good_for_the_row_it_deleted()
{
fx.SkipIfUnavailable();
var rig = MakeRig(fx);
await rig.Updater.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T0, 1)], default);
// +7 ticks is inside the same µs; the row lives at T0.
(await rig.Updater.DeleteAtTimeAsync("t", [T0.AddTicks(7)], default)).ShouldBe([Good]);
(await RowsAsync(rig, "t", liveOnly: true)).ShouldBeEmpty();
}
[Fact]
public async Task DeleteRaw_is_half_open_pages_through_big_ranges_and_reports_BadNoData_when_empty()
{
fx.SkipIfUnavailable();
var rig = MakeRig(fx);
(await rig.Updater.DeleteRawAsync("unknown", T0, T0.AddHours(1), default)).ShouldBe(HistorianStatusCodes.BadNoData);
// 2,500 rows => 3 pages of at most 1,000; plus one row exactly at the (exclusive) end.
var values = Enumerable.Range(0, 2500).Select(i => V(T0.AddSeconds(i), i)).ToList();
values.Add(V(T0.AddSeconds(2500), -1));
(await rig.Updater.UpdateDataAsync("big", HistoryUpdateMode.Insert, values, default))
.ShouldAllBe(c => c == HistorianStatusCodes.GoodEntryInserted);
(await rig.Updater.DeleteRawAsync("big", T0, T0.AddSeconds(2500), default)).ShouldBe(Good);
var live = await RowsAsync(rig, "big", liveOnly: true);
live.Count.ShouldBe(1);
live[0].Num.ShouldBe(-1d);
(await rig.Updater.DeleteRawAsync("big", T0, T0.AddSeconds(2500), default)).ShouldBe(HistorianStatusCodes.BadNoData);
}
[Fact]
public async Task DeleteRaw_with_a_pre_1970_start_reads_from_the_bound_unclamped()
{
fx.SkipIfUnavailable();
var rig = MakeRig(fx);
await rig.Updater.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T0, 1), V(T0.AddSeconds(1), 2)], default);
(await rig.Updater.DeleteRawAsync("t", new DateTime(1900, 1, 1, 0, 0, 0, DateTimeKind.Utc), T0.AddSeconds(1), default))
.ShouldBe(Good);
(await RowsAsync(rig, "t", liveOnly: true)).Select(r => r.Num).ShouldBe([2d]);
// DateTime.MinValue as the start is nudged inside, not bound as -infinity
(await rig.Updater.DeleteRawAsync("t", DateTime.MinValue, DateTime.MaxValue, default)).ShouldBe(Good);
(await RowsAsync(rig, "t", liveOnly: true)).ShouldBeEmpty();
}
[Fact]
public async Task Read_after_write_is_visible_across_many_immediate_calls()
{
fx.SkipIfUnavailable();
var rig = MakeRig(fx);
// ~15% of un-waited inserts are not yet visible (spike Q4), so 40 round trips expose a missing wait.
for (var i = 0; i < 40; i++)
{
var t = T0.AddSeconds(i);
(await rig.Updater.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(t, i)], default))
.ShouldBe([HistorianStatusCodes.GoodEntryInserted]);
(await rig.Updater.UpdateDataAsync("t", HistoryUpdateMode.Replace, [V(t, i + 1000)], default))
.ShouldBe([HistorianStatusCodes.GoodEntryReplaced], $"iteration {i}: the insert was not visible");
}
}
[Fact]
public async Task Concurrent_inserts_of_the_same_new_key_yield_exactly_one_GoodEntryInserted()
{
fx.SkipIfUnavailable();
var rig = MakeRig(fx);
for (var i = 0; i < 20; i++) // loop so the race shows reliably without the lock
{
var t = T0.AddSeconds(i);
var codes = await Task.WhenAll(
Task.Run(() => rig.Updater.UpdateDataAsync("race", HistoryUpdateMode.Insert, [V(t, 1)], default)),
Task.Run(() => rig.Updater.UpdateDataAsync("race", HistoryUpdateMode.Insert, [V(t, 2)], default)));
codes.Select(c => c[0]).OrderBy(c => c).ToArray().ShouldBe(
new[] { HistorianStatusCodes.GoodEntryInserted, HistorianStatusCodes.BadEntryExists }, $"iteration {i}");
}
}
[Fact]
public async Task Existence_reads_are_bounded_when_request_times_are_far_apart()
{
fx.SkipIfUnavailable();
var rig = MakeRig(fx);
// 1 Hz seed across 2 h = 7,200 live rows; the request touches 3 instants 1 h apart.
var seed = Enumerable.Range(0, 7200).Select(i => V(T0.AddSeconds(i), i)).ToList();
await rig.Updater.UpdateDataAsync("sparse", HistoryUpdateMode.Insert, seed, default);
var before = rig.Updater.LiveRowsRead;
var codes = await rig.Updater.UpdateDataAsync("sparse", HistoryUpdateMode.Replace,
[V(T0, 1), V(T0.AddHours(1), 2), V(T0.AddHours(2).AddSeconds(-1), 3), V(T0.AddHours(1).AddMilliseconds(500), 4)], default);
codes.ShouldBe([HistorianStatusCodes.GoodEntryReplaced, HistorianStatusCodes.GoodEntryReplaced,
HistorianStatusCodes.GoodEntryReplaced, HistorianStatusCodes.BadNoEntryExists]);
// 3 clusters of a single instant each (T0+1h and +1h.5s are one cluster of 2 instants, 1 live row)
(rig.Updater.LiveRowsRead - before).ShouldBeLessThanOrEqualTo(3);
}
}
tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/WriteTests.cs
+277
View File
@@ -0,0 +1,277 @@
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests;
[Collection(QuestDbCollection.Name)]
public sealed class WriteTests(QuestDbFixture fx)
{
// µs-aligned (ticks % 10 == 0) so "same µs" offsets of a few ticks stay inside one 10-tick window.
private static readonly DateTime T0 = new(2026, 10, 9, 1, 2, 3, DateTimeKind.Utc);
private static readonly DateTime ServerNow = new(2026, 10, 9, 12, 0, 0, DateTimeKind.Utc);
private (QuestDbHistorianValueWriter W, QuestDbConnection C) Make(string? prefix = null, string? connectionString = null)
{
var o = fx.Options(prefix ?? fx.UniquePrefix());
if (connectionString is not null) o.ConnectionString = connectionString;
var c = new QuestDbConnection(o);
var b = new QuestDbSchemaBootstrapper(c, o);
var wal = new QuestDbWalVisibility(c, o.VisibilityTimeout);
return (new QuestDbHistorianValueWriter(c, new QuestDbTagCatalog(c, b, wal), wal, utcNow: () => ServerNow), c);
}
private static async Task<List<(DateTime T, byte Kind, double? N, long? I, string? S, long Status, bool Deleted, DateTime ServerTs)>> RowsAsync(
QuestDbConnection c, string tag)
{
await using var conn = await c.OpenAsync(default);
await using var cmd = c.CreateCommand(conn,
$"SELECT ts, kind, num_value, int_value, str_value, status, deleted, server_ts FROM {c.SampleTable} WHERE tag = $1 ORDER BY ts");
QuestDbConnection.BindText(cmd, tag);
var l = new List<(DateTime, byte, double?, long?, string?, long, bool, DateTime)>();
await using var r = await cmd.ExecuteReaderAsync();
while (await r.ReadAsync())
l.Add((QuestDbConnection.ReadUtc(r, 0), QuestDbConnection.ReadKind(r, 1), r.IsDBNull(2) ? null : r.GetDouble(2),
r.IsDBNull(3) ? null : r.GetInt64(3), r.IsDBNull(4) ? null : r.GetString(4), r.GetInt64(5), r.GetBoolean(6),
QuestDbConnection.ReadUtc(r, 7)));
return l;
}
private static async Task SeedTombstoneAsync(QuestDbConnection c, QuestDbWalVisibility wal, string tag, DateTime t)
{
await using (var conn = await c.OpenAsync(default))
{
await using var cmd = c.CreateCommand(conn,
$"INSERT INTO {c.SampleTable} (ts, tag, kind, num_value, int_value, str_value, status, deleted, server_ts) VALUES ($1,$2,$3,NULL,NULL,NULL,$4,true,$5)");
QuestDbConnection.BindTimestamp(cmd, t);
QuestDbConnection.BindText(cmd, tag);
QuestDbConnection.BindByte(cmd, 0);
QuestDbConnection.BindLong(cmd, 0);
QuestDbConnection.BindTimestamp(cmd, t);
await cmd.ExecuteNonQueryAsync();
}
await wal.WaitAsync(c.SampleTable, default);
}
[Fact]
public async Task All_six_kinds_round_trip_with_a_non_Good_status()
{
fx.SkipIfUnavailable();
var (w, c) = Make();
const uint uncertain = HistorianStatusCodes.UncertainDataSubNormal;
const uint bad = HistorianStatusCodes.BadOutOfRange;
var dt = new DateTime(2025, 5, 6, 7, 8, 9, DateTimeKind.Utc).AddTicks(1230);
HistorizationValue[] vals =
[
new(T0, HistorianValue.FromDouble(1.5), uncertain),
new(T0.AddSeconds(1), HistorianValue.FromInt64(-77), uncertain),
new(T0.AddSeconds(2), HistorianValue.FromBoolean(true), uncertain),
new(T0.AddSeconds(3), HistorianValue.FromString("héllo"), uncertain),
new(T0.AddSeconds(4), HistorianValue.FromDateTime(dt), uncertain),
new(T0.AddSeconds(5), HistorianValue.Empty, bad),
];
(await w.WriteLiveValuesAsync("tag.kinds", vals, CancellationToken.None)).ShouldBeTrue();
var rows = await RowsAsync(c, "tag.kinds");
rows.Count.ShouldBe(6);
for (var i = 0; i < 6; i++)
{
rows[i].T.ShouldBe(vals[i].TimestampUtc!.Value);
rows[i].Status.ShouldBe(SampleColumnMapper.ToLongStatus(vals[i].StatusCode));
rows[i].Deleted.ShouldBeFalse();
rows[i].ServerTs.ShouldBe(ServerNow);
SampleColumnMapper.FromColumns(rows[i].Kind, rows[i].N, rows[i].I, rows[i].S).ShouldBe(vals[i].Value);
}
}
[Fact]
public async Task A_redelivered_batch_leaves_one_row_and_the_first_value()
{
fx.SkipIfUnavailable();
var (w, c) = Make();
(await w.WriteLiveValuesAsync("tag.redeliver", [new(T0, HistorianValue.FromDouble(1), 0)], CancellationToken.None)).ShouldBeTrue();
(await w.WriteLiveValuesAsync("tag.redeliver", [new(T0, HistorianValue.FromDouble(2), 0)], CancellationToken.None)).ShouldBeTrue();
var rows = await RowsAsync(c, "tag.redeliver");
rows.Count.ShouldBe(1);
rows[0].N.ShouldBe(1d);
}
[Fact]
public async Task Redelivery_straight_after_the_write_always_sees_it_because_the_writer_waits_for_WAL_visibility()
{
fx.SkipIfUnavailable();
var (w, c) = Make();
// Without the visibility wait ~35 % of immediate re-reads miss the row (T0 Q4) and the second value would replace the first.
for (var i = 0; i < 40; i++)
{
var t = T0.AddSeconds(i);
(await w.WriteLiveValuesAsync("tag.rw", [new(t, HistorianValue.FromDouble(1), 0)], CancellationToken.None)).ShouldBeTrue();
(await w.WriteLiveValuesAsync("tag.rw", [new(t, HistorianValue.FromDouble(2), 0)], CancellationToken.None)).ShouldBeTrue();
}
var rows = await RowsAsync(c, "tag.rw");
rows.Count.ShouldBe(40);
rows.ShouldAllBe(r => r.N == 1d);
}
[Fact]
public async Task A_tombstoned_key_is_not_resurrected_by_redelivery()
{
fx.SkipIfUnavailable();
var (w, c) = Make();
var wal = new QuestDbWalVisibility(c, TimeSpan.FromSeconds(5));
(await w.WriteLiveValuesAsync("tag.tomb", [new(T0, HistorianValue.FromDouble(9), 0)], CancellationToken.None)).ShouldBeTrue();
await SeedTombstoneAsync(c, wal, "tag.tomb", T0.AddSeconds(1));
// Redeliver the tombstoned key plus a fresh neighbour.
(await w.WriteLiveValuesAsync("tag.tomb",
[new(T0.AddSeconds(1), HistorianValue.FromDouble(5), 0), new(T0.AddSeconds(2), HistorianValue.FromDouble(6), 0)],
CancellationToken.None)).ShouldBeTrue();
var rows = await RowsAsync(c, "tag.tomb");
rows.Count.ShouldBe(3);
rows.Single(r => r.T == T0.AddSeconds(1)).Deleted.ShouldBeTrue();
rows.Single(r => r.T == T0.AddSeconds(2)).N.ShouldBe(6d);
}
[Fact]
public async Task NaN_is_stored_as_Empty_with_BadOutOfRange()
{
fx.SkipIfUnavailable();
var (w, c) = Make();
(await w.WriteLiveValuesAsync("tag.nan", [new(T0, HistorianValue.FromDouble(double.NaN), 0)], CancellationToken.None)).ShouldBeTrue();
var rows = await RowsAsync(c, "tag.nan");
rows.Count.ShouldBe(1);
rows[0].Kind.ShouldBe((byte)HistorianValueKind.Empty);
rows[0].Status.ShouldBe((long)HistorianStatusCodes.BadOutOfRange);
rows[0].N.ShouldBeNull();
}
[Fact]
public async Task A_microsecond_collision_within_a_batch_keeps_the_first()
{
fx.SkipIfUnavailable();
var (w, c) = Make();
// +2 and +7 ticks stay inside one 10-tick (1 µs) window of the aligned T0.
(await w.WriteLiveValuesAsync("tag.us",
[new(T0.AddTicks(2), HistorianValue.FromDouble(1), 0), new(T0.AddTicks(7), HistorianValue.FromDouble(2), 0)],
CancellationToken.None)).ShouldBeTrue();
var rows = await RowsAsync(c, "tag.us");
rows.Count.ShouldBe(1);
rows[0].N.ShouldBe(1d);
rows[0].T.ShouldBe(T0);
}
[Fact]
public async Task Samples_before_1970_are_dropped_and_the_rest_still_land()
{
fx.SkipIfUnavailable();
var (w, c) = Make();
var old = new DateTime(1960, 1, 1, 0, 0, 0, DateTimeKind.Utc);
(await w.WriteLiveValuesAsync("tag.old",
[new(old, HistorianValue.FromDouble(1), 0), new(T0, HistorianValue.FromDouble(2), 0)], CancellationToken.None)).ShouldBeTrue();
w.DroppedPre1970.ShouldBe(1);
var rows = await RowsAsync(c, "tag.old");
rows.Count.ShouldBe(1);
rows[0].T.ShouldBe(T0);
// A batch of nothing but unstorable samples succeeds (nothing to retry) and stores nothing.
(await w.WriteLiveValuesAsync("tag.old2", [new(old, HistorianValue.FromDouble(1), 0)], CancellationToken.None)).ShouldBeTrue();
w.DroppedPre1970.ShouldBe(2);
(await RowsAsync(c, "tag.old2")).ShouldBeEmpty();
}
[Fact]
public async Task A_batch_larger_than_one_insert_chunk_lands_completely()
{
fx.SkipIfUnavailable();
var (w, c) = Make();
var vals = Enumerable.Range(0, 2500).Select(i => new HistorizationValue(T0.AddMilliseconds(i), HistorianValue.FromInt64(i), 0)).ToArray();
(await w.WriteLiveValuesAsync("tag.big", vals, CancellationToken.None)).ShouldBeTrue();
var rows = await RowsAsync(c, "tag.big");
rows.Count.ShouldBe(2500);
rows[1999].I.ShouldBe(1999);
}
[Fact]
public async Task An_over_long_tag_name_returns_false()
{
fx.SkipIfUnavailable();
var (w, _) = Make();
(await w.WriteLiveValuesAsync(new string('x', 401), [new(T0, HistorianValue.FromDouble(1), 0)], CancellationToken.None)).ShouldBeFalse();
}
[Fact]
public async Task A_stopped_port_returns_false_without_throwing()
{
fx.SkipIfUnavailable();
var (w, _) = Make(connectionString: "Host=localhost;Port=1;Username=admin;Password=quest;Database=qdb;ServerCompatibilityMode=NoTypeLoading;Timeout=2");
(await w.WriteLiveValuesAsync("x", [new(T0, HistorianValue.FromDouble(1), 0)], CancellationToken.None)).ShouldBeFalse();
}
[Fact]
public async Task A_cancelled_write_returns_false_without_throwing()
{
fx.SkipIfUnavailable();
var (w, _) = Make();
using var cts = new CancellationTokenSource();
cts.Cancel();
(await w.WriteLiveValuesAsync("x", [new(T0, HistorianValue.FromDouble(1), 0)], cts.Token)).ShouldBeFalse();
}
private static async Task SeedRowAsync(QuestDbConnection c, QuestDbWalVisibility wal, string tag, DateTime t, double v)
{
await using (var conn = await c.OpenAsync(default))
{
await using var cmd = c.CreateCommand(conn,
$"INSERT INTO {c.SampleTable} (ts, tag, kind, num_value, int_value, str_value, status, deleted, server_ts) VALUES ($1,$2,1,$3,NULL,NULL,0,false,$4)");
QuestDbConnection.BindTimestamp(cmd, t);
QuestDbConnection.BindText(cmd, tag);
cmd.Parameters.Add(new Npgsql.NpgsqlParameter { NpgsqlDbType = NpgsqlTypes.NpgsqlDbType.Double, Value = v });
QuestDbConnection.BindTimestamp(cmd, t);
await cmd.ExecuteNonQueryAsync();
}
await wal.WaitAsync(c.SampleTable, default);
}
[Fact]
public async Task A_row_on_another_tag_at_the_same_timestamp_does_not_suppress_the_write()
{
fx.SkipIfUnavailable();
var (w, c) = Make();
var wal = new QuestDbWalVisibility(c, TimeSpan.FromSeconds(5));
(await w.WriteLiveValuesAsync("tag.A", [new(T0, HistorianValue.FromDouble(1), 0)], CancellationToken.None)).ShouldBeTrue();
(await w.WriteLiveValuesAsync("tag.B", [new(T0, HistorianValue.FromDouble(2), 0)], CancellationToken.None)).ShouldBeTrue();
(await RowsAsync(c, "tag.A")).Single().N.ShouldBe(1d);
(await RowsAsync(c, "tag.B")).Single().N.ShouldBe(2d);
}
[Fact]
public async Task An_absent_interior_timestamp_is_written_while_both_present_ends_are_left_alone()
{
fx.SkipIfUnavailable();
var (w, c) = Make();
var wal = new QuestDbWalVisibility(c, TimeSpan.FromSeconds(5));
await w.WriteLiveValuesAsync("tag.ends", [new(T0.AddSeconds(100), HistorianValue.FromDouble(0), 0)], CancellationToken.None); // creates the tag/tables
await SeedRowAsync(c, wal, "tag.ends", T0, 10);
await SeedRowAsync(c, wal, "tag.ends", T0.AddSeconds(20), 30);
(await w.WriteLiveValuesAsync("tag.ends",
[new(T0, HistorianValue.FromDouble(-1), 0), new(T0.AddSeconds(10), HistorianValue.FromDouble(20), 0), new(T0.AddSeconds(20), HistorianValue.FromDouble(-3), 0)],
CancellationToken.None)).ShouldBeTrue();
var rows = (await RowsAsync(c, "tag.ends")).Where(r => r.T <= T0.AddSeconds(20)).ToList();
rows.Select(r => r.N).ShouldBe([10d, 20d, 30d]);
}
[Fact]
public async Task An_outlier_timestamp_does_not_widen_the_existence_read_over_the_stored_rows_between()
{
fx.SkipIfUnavailable();
var (w, c) = Make();
// Behavioural check of the whole path with a far-apart batch: both samples land, a redelivery adds nothing.
var far = T0.AddYears(1);
HistorizationValue[] vals = [new(T0, HistorianValue.FromDouble(1), 0), new(far, HistorianValue.FromDouble(2), 0)];
(await w.WriteLiveValuesAsync("tag.far", vals, CancellationToken.None)).ShouldBeTrue();
(await w.WriteLiveValuesAsync("tag.far", vals, CancellationToken.None)).ShouldBeTrue();
(await RowsAsync(c, "tag.far")).Count.ShouldBe(2);
}
}
tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests.csproj
+33
View File
@@ -0,0 +1,33 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>net10.0</TargetFramework>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
<IsPackable>false</IsPackable>
<IsTestProject>true</IsTestProject>
<RootNamespace>ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests</RootNamespace>
<NoWarn>$(NoWarn);xUnit1051</NoWarn>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="xunit.v3"/>
<PackageReference Include="Shouldly"/>
<PackageReference Include="Microsoft.NET.Test.Sdk"/>
<PackageReference Include="xunit.runner.visualstudio">
<PrivateAssets>all</PrivateAssets>
<IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
</PackageReference>
</ItemGroup>
<ItemGroup>
<ProjectReference Include="..\..\..\src\Drivers\ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb\ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.csproj"/>
<ProjectReference Include="..\..\..\src\Drivers\ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common\ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.csproj"/>
</ItemGroup>
<ItemGroup>
<!-- Shared conformance suite (owned by the Conformance task); an empty glob until it lands. -->
<Compile Include="..\Historian.Conformance\**\*.cs" LinkBase="Conformance"/>
</ItemGroup>
</Project>
tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests/QuestDbConnectionTests.cs
+184
View File
@@ -0,0 +1,184 @@
using System.Net.Sockets;
using Npgsql;
using NpgsqlTypes;
using Shouldly;
using Xunit;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests;
public sealed class QuestDbConnectionTests
{
private static DateTime Bound(DateTime v, bool clamp = false)
{
var cmd = new NpgsqlCommand();
var p = QuestDbConnection.BindTimestamp(cmd, v, clamp);
p.NpgsqlDbType.ShouldBe(NpgsqlDbType.TimestampTz);
var bound = (DateTime)p.Value!;
bound.Kind.ShouldBe(DateTimeKind.Utc);
return bound;
}
[Fact]
public void Round_truncates_sub_microsecond_ticks()
{
var t = new DateTime(2026, 3, 1, 12, 0, 0, DateTimeKind.Utc).AddTicks(1234567);
QuestDbConnection.Round(t).ShouldBe(new DateTime(2026, 3, 1, 12, 0, 0, DateTimeKind.Utc).AddTicks(1234560));
}
[Fact]
public void Round_truncates_rather_than_rounds_before_2000()
{
var t = new DateTime(1990, 5, 5, 0, 0, 0, DateTimeKind.Utc).AddTicks(1234565);
QuestDbConnection.Round(t).Ticks.ShouldBe(t.Ticks - 5);
}
[Fact]
public void Round_treats_Unspecified_as_UTC()
{
var u = new DateTime(2026, 3, 1, 12, 0, 0, DateTimeKind.Unspecified);
var r = QuestDbConnection.Round(u);
r.Kind.ShouldBe(DateTimeKind.Utc);
r.Ticks.ShouldBe(u.Ticks);
}
[Fact]
public void Round_converts_Local_to_UTC()
{
var local = new DateTime(2026, 3, 1, 12, 0, 0, DateTimeKind.Local);
var r = QuestDbConnection.Round(local);
r.Kind.ShouldBe(DateTimeKind.Utc);
r.ShouldBe(local.ToUniversalTime());
}
[Fact]
public void Round_keeps_a_UTC_value_that_is_already_aligned()
{
var t = new DateTime(2026, 3, 1, 12, 0, 0, DateTimeKind.Utc);
QuestDbConnection.Round(t).ShouldBe(t);
}
[Theory]
[InlineData(0)] // MinValue itself
[InlineData(5)] // truncates to MinValue
public void BindTimestamp_never_emits_MinValue(long ticks)
{
var b = Bound(new DateTime(ticks, DateTimeKind.Utc));
b.ShouldNotBe(DateTime.MinValue);
b.Ticks.ShouldBe(10);
b.ShouldBe(new DateTime(10, DateTimeKind.Utc)); // 0001-01-01T00:00:00.000001 (T0-confirmed)
}
[Fact]
public void BindTimestamp_never_emits_MaxValue()
{
var b = Bound(DateTime.MaxValue);
b.ShouldNotBe(DateTime.MaxValue);
b.ShouldBe(new DateTime(9999, 12, 31, 23, 59, 59, DateTimeKind.Utc).AddTicks(9999990)); // T0-confirmed bound
Bound(DateTime.MinValue).ShouldNotBe(DateTime.MinValue);
}
[Fact]
public void BindTimestamp_never_emits_either_extreme_for_any_kind()
{
foreach (var kind in new[] { DateTimeKind.Utc, DateTimeKind.Unspecified, DateTimeKind.Local })
foreach (var t in new[] { new DateTime(DateTime.MinValue.Ticks, kind), new DateTime(DateTime.MaxValue.Ticks, kind) })
{
var b = Bound(t);
b.ShouldNotBe(DateTime.MinValue);
b.ShouldNotBe(DateTime.MaxValue);
}
}
[Fact]
public void BindTimestamp_keeps_a_pre_1970_range_bound_unless_asked_to_clamp()
{
var pre = new DateTime(1960, 1, 1, 0, 0, 0, DateTimeKind.Utc);
Bound(pre).ShouldBe(pre);
Bound(pre, clamp: true).ShouldBe(DateTime.UnixEpoch);
Bound(new DateTime(2026, 1, 1, 0, 0, 0, DateTimeKind.Utc), clamp: true).Year.ShouldBe(2026);
}
[Fact]
public void IsStorable_is_false_only_before_1970()
{
QuestDbConnection.IsStorable(DateTime.UnixEpoch).ShouldBeTrue();
QuestDbConnection.IsStorable(DateTime.UnixEpoch.AddTicks(-1)).ShouldBeFalse();
QuestDbConnection.IsStorable(new DateTime(1601, 1, 1, 0, 0, 0, DateTimeKind.Utc)).ShouldBeFalse();
QuestDbConnection.IsStorable(DateTime.MinValue).ShouldBeFalse();
QuestDbConnection.IsStorable(DateTime.MaxValue).ShouldBeTrue();
QuestDbConnection.IsStorable(new DateTime(2026, 1, 1, 0, 0, 0, DateTimeKind.Utc)).ShouldBeTrue();
}
[Fact]
public void BindByte_binds_smallint()
{
var p = QuestDbConnection.BindByte(new NpgsqlCommand(), 200);
p.NpgsqlDbType.ShouldBe(NpgsqlDbType.Smallint);
p.Value.ShouldBe((short)200);
}
[Fact]
public void IsTransient_classifies_transport_failures_not_server_rejections()
{
QuestDbConnection.IsTransient(new SocketException()).ShouldBeTrue();
QuestDbConnection.IsTransient(new IOException("x")).ShouldBeTrue();
QuestDbConnection.IsTransient(new TimeoutException()).ShouldBeTrue();
QuestDbConnection.IsTransient(new NpgsqlException("x", new IOException())).ShouldBeTrue();
QuestDbConnection.IsTransient(new InvalidOperationException("x")).ShouldBeFalse();
QuestDbConnection.IsTransient(new OperationCanceledException()).ShouldBeFalse();
}
[Fact]
public void T_prefixes_the_table_base_name()
{
var c = new QuestDbConnection(new QuestDbHistorianOptions { ConnectionString = "Host=127.0.0.1;Port=1", TablePrefix = "p_" });
c.T("sample").ShouldBe("p_sample");
c.SampleTable.ShouldBe("p_sample");
c.EventTable.ShouldBe("p_event");
c.TagTable.ShouldBe("p_tag");
}
// ---- pool size (final review I-1) -------------------------------------------------------------------------------
private static int PoolOf(string cs)
{
using var c = new QuestDbConnection(new QuestDbHistorianOptions { ConnectionString = cs });
return c.EffectiveMaxPoolSize;
}
[Fact]
public void An_unset_Maximum_Pool_Size_is_capped_at_24() =>
PoolOf("Host=127.0.0.1;Port=1").ShouldBe(24);
[Fact]
public void The_default_cap_constant_is_24() => QuestDbConnection.DefaultMaxPoolSize.ShouldBe(24);
[Theory]
[InlineData("Host=127.0.0.1;Port=1;Maximum Pool Size=10", 10)]
[InlineData("Host=127.0.0.1;Port=1;Maximum Pool Size=80", 80)]
[InlineData("Host=127.0.0.1;Port=1;maximum pool size=30", 30)]
[InlineData("Host=127.0.0.1;Port=1;MaxPoolSize=80", 80)]
public void An_explicit_Maximum_Pool_Size_is_preserved(string cs, int expected) =>
PoolOf(cs).ShouldBe(expected);
[Fact]
public void The_effective_pool_size_is_logged_once_at_Information()
{
var log = new InfoLogger();
using var c = new QuestDbConnection(new QuestDbHistorianOptions { ConnectionString = "Host=127.0.0.1;Port=1" }, log);
log.Messages.Count.ShouldBe(1);
log.Messages[0].ShouldContain("24");
}
private sealed class InfoLogger : Microsoft.Extensions.Logging.ILogger
{
public List<string> Messages { get; } = [];
public IDisposable? BeginScope<TState>(TState state) where TState : notnull => null;
public bool IsEnabled(Microsoft.Extensions.Logging.LogLevel logLevel) => true;
public void Log<TState>(Microsoft.Extensions.Logging.LogLevel logLevel, Microsoft.Extensions.Logging.EventId eventId, TState state,
Exception? exception, Func<TState, Exception?, string> formatter)
{
if (logLevel == Microsoft.Extensions.Logging.LogLevel.Information) Messages.Add(formatter(state, exception));
}
}
}
tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests/QuestDbHistorianOptionsTests.cs
+79
View File
@@ -0,0 +1,79 @@
using Shouldly;
using Xunit;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests;
public sealed class QuestDbHistorianOptionsTests
{
private static QuestDbHistorianOptions Valid() => new() { ConnectionString = "Host=h;Port=8812" };
[Fact]
public void Defaults_are_the_documented_ones()
{
var o = new QuestDbHistorianOptions();
o.ConnectionString.ShouldBe("");
o.TablePrefix.ShouldBe("hist_");
o.RetentionDays.ShouldBeNull();
o.CommandTimeout.ShouldBe(TimeSpan.FromSeconds(30));
o.AutoCreateSchema.ShouldBeTrue();
o.VisibilityTimeout.ShouldBe(TimeSpan.FromSeconds(5));
}
[Fact]
public void A_valid_configuration_has_no_errors() => Valid().Validate().ShouldBeEmpty();
[Fact]
public void Missing_connection_string_names_the_key()
{
var o = Valid(); o.ConnectionString = " ";
o.Validate().ShouldHaveSingleItem().ShouldContain("ServerHistorian:QuestDb:ConnectionString");
}
[Theory]
[InlineData("")]
[InlineData("hist-")]
[InlineData("hist.")]
[InlineData("hist_; DROP")]
public void Bad_prefix_names_the_key(string prefix)
{
var o = Valid(); o.TablePrefix = prefix;
o.Validate().ShouldHaveSingleItem().ShouldContain("ServerHistorian:QuestDb:TablePrefix");
}
[Theory]
[InlineData("t0123456789ab_")]
[InlineData("A_9")]
public void Good_prefixes_pass(string prefix)
{
var o = Valid(); o.TablePrefix = prefix;
o.Validate().ShouldBeEmpty();
}
[Theory]
[InlineData(0)]
[InlineData(-3)]
public void Retention_below_one_names_the_key(int days)
{
var o = Valid(); o.RetentionDays = days;
o.Validate().ShouldHaveSingleItem().ShouldContain("ServerHistorian:QuestDb:RetentionDays");
}
[Fact]
public void Retention_of_one_is_valid() { var o = Valid(); o.RetentionDays = 1; o.Validate().ShouldBeEmpty(); }
[Fact]
public void Non_positive_command_timeout_names_the_key()
{
var o = Valid(); o.CommandTimeout = TimeSpan.Zero;
o.Validate().ShouldHaveSingleItem().ShouldContain("ServerHistorian:QuestDb:CommandTimeout");
}
[Fact]
public void Negative_visibility_timeout_names_the_key_and_zero_is_valid()
{
var o = Valid(); o.VisibilityTimeout = TimeSpan.FromMilliseconds(-1);
o.Validate().ShouldHaveSingleItem().ShouldContain("ServerHistorian:QuestDb:VisibilityTimeout");
o.VisibilityTimeout = TimeSpan.Zero;
o.Validate().ShouldBeEmpty();
}
}
tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests/QuestDbHistorianProviderTests.cs
+107
View File
@@ -0,0 +1,107 @@
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests;
public sealed class QuestDbHistorianProviderTests
{
private static QuestDbHistorianProvider Make() => new(new QuestDbHistorianOptions
{
ConnectionString = "Host=localhost;Port=18812;Username=admin;Password=quest;Database=qdb;ServerCompatibilityMode=NoTypeLoading",
});
[Fact]
public void Name_and_all_six_kinds()
{
var p = Make();
p.Name.ShouldBe("QuestDb");
p.SupportedKinds.ShouldBe(Enum.GetValues<HistorianValueKind>(), ignoreOrder: true);
p.SupportedKinds.Count.ShouldBe(6);
}
[Fact]
public void Creates_all_five_seams_without_touching_questdb()
{
var sp = new NullSp();
var p = Make();
p.CreateDataSource(sp).ShouldBeOfType<QuestDbHistorianDataSource>();
p.CreateValueWriter(sp).ShouldBeOfType<QuestDbHistorianValueWriter>();
p.CreateAlarmWriter(sp).ShouldBeOfType<QuestDbAlarmHistorianWriter>();
p.CreateProvisioning(sp).ShouldBeOfType<QuestDbTagProvisioner>();
p.CreateUpdater(sp).ShouldBeOfType<QuestDbHistorianDataUpdater>();
}
[Fact]
public void Supported_kinds_are_a_frozen_set() =>
Make().SupportedKinds.ShouldBeAssignableTo<System.Collections.Frozen.FrozenSet<HistorianValueKind>>();
/// <summary>
/// An invalid table prefix must not fail provider construction (the host resolves the provider even when no
/// consumer is enabled); it surfaces only when a seam is actually built.
/// </summary>
[Fact]
public void An_invalid_prefix_fails_only_when_a_seam_is_built()
{
var p = new QuestDbHistorianProvider(new QuestDbHistorianOptions { ConnectionString = "Host=x", TablePrefix = "bad;prefix" });
p.Name.ShouldBe("QuestDb");
Should.Throw<ArgumentException>(() => p.CreateDataSource(new NullSp()));
}
[Fact]
public void Every_seam_shares_one_connection_bootstrapper_catalog_and_wal_wait()
{
var sp = new NullSp();
var p = Make();
var a = (QuestDbHistorianDataSource)p.CreateDataSource(sp);
var b = (QuestDbHistorianDataSource)p.CreateDataSource(sp);
var w = p.CreateValueWriter(sp);
var u = p.CreateUpdater(sp)!;
var prov = p.CreateProvisioning(sp);
var alarm = p.CreateAlarmWriter(sp);
Field(a, "_conn").ShouldBeSameAs(Field(b, "_conn"));
Field(a, "_conn").ShouldBeSameAs(Field(w, "_conn"));
Field(a, "_conn").ShouldBeSameAs(Field(u, "_conn"));
Field(a, "_conn").ShouldBeSameAs(Field(alarm, "_conn"));
Field(a, "_bootstrap").ShouldBeSameAs(Field(b, "_bootstrap"));
Field(a, "_bootstrap").ShouldBeSameAs(Field(prov, "_bootstrap"));
Field(a, "_bootstrap").ShouldBeSameAs(Field(alarm, "_bootstrap"));
Field(a, "_catalog").ShouldBeSameAs(Field(b, "_catalog"));
Field(a, "_catalog").ShouldBeSameAs(Field(w, "_catalog"));
Field(a, "_catalog").ShouldBeSameAs(Field(u, "_tags"));
Field(a, "_catalog").ShouldBeSameAs(Field(prov, "_catalog"));
Field(w, "_wal").ShouldBeSameAs(Field(u, "_wal"));
Field(w, "_wal").ShouldBeSameAs(Field(prov, "_wal"));
Field(w, "_wal").ShouldBeSameAs(Field(alarm, "_wal"));
}
[Fact]
public async Task Disposing_the_provider_disposes_the_shared_pool_and_is_idempotent()
{
var sp = new NullSp();
var p = Make();
var conn = (QuestDbConnection)Field(p.CreateDataSource(sp), "_conn")!;
var pool = (Npgsql.NpgsqlDataSource)Field(conn, "_dataSource")!;
await p.DisposeAsync();
Should.Throw<ObjectDisposedException>(() => pool.CreateConnection().Open());
await p.DisposeAsync();
p.Dispose();
Should.Throw<ObjectDisposedException>(() => p.CreateDataSource(sp));
}
[Fact]
public async Task Disposing_an_unused_provider_is_a_no_op() => await Make().DisposeAsync();
[Fact]
public void The_constructor_rejects_null_options() =>
Should.Throw<ArgumentNullException>(() => new QuestDbHistorianProvider(null!));
private static object? Field(object o, string name) =>
o.GetType().GetField(name, System.Reflection.BindingFlags.Instance | System.Reflection.BindingFlags.NonPublic)!.GetValue(o);
private sealed class NullSp : IServiceProvider
{
public object? GetService(Type serviceType) => null;
}
}
tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests/QuestDbHistorianValueWriterTests.cs
+71
View File
@@ -0,0 +1,71 @@
using Microsoft.Extensions.Logging;
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests;
/// <summary>
/// The pre-1970 drop Warning is rate-limited per tag (final review M-3) while the counter still counts every dropped
/// sample. All-pre-1970 batches return before any QuestDB call, so the connection points at a closed port.
/// </summary>
public sealed class QuestDbHistorianValueWriterTests
{
private static readonly DateTime FileTimeZero = new(1601, 1, 1, 0, 0, 0, DateTimeKind.Utc);
private static (QuestDbHistorianValueWriter W, ListLogger Log, Action<TimeSpan> Advance) Make()
{
var opts = new QuestDbHistorianOptions { ConnectionString = "Host=127.0.0.1;Port=1;Timeout=1" };
var conn = new QuestDbConnection(opts);
var boot = new QuestDbSchemaBootstrapper(conn, opts);
var wal = new QuestDbWalVisibility(conn, opts.VisibilityTimeout);
var now = new DateTime(2026, 10, 11, 12, 0, 0, DateTimeKind.Utc);
var log = new ListLogger();
var w = new QuestDbHistorianValueWriter(conn, new QuestDbTagCatalog(conn, boot, wal), wal, log, () => now);
return (w, log, d => now += d);
}
private static HistorizationValue[] Pre1970(int n) =>
Enumerable.Range(0, n).Select(i => new HistorizationValue(FileTimeZero.AddSeconds(i), HistorianValue.FromDouble(i), 0)).ToArray();
[Fact]
public async Task The_pre_1970_warning_is_logged_once_per_tag_per_interval_but_every_drop_is_counted()
{
var (w, log, advance) = Make();
(await w.WriteLiveValuesAsync("a", Pre1970(2), CancellationToken.None)).ShouldBeTrue();
advance(TimeSpan.FromSeconds(2));
(await w.WriteLiveValuesAsync("a", Pre1970(3), CancellationToken.None)).ShouldBeTrue();
advance(QuestDbHistorianValueWriter.Pre1970WarningInterval - TimeSpan.FromSeconds(3));
(await w.WriteLiveValuesAsync("a", Pre1970(1), CancellationToken.None)).ShouldBeTrue();
log.Warnings.ShouldBe(1);
w.DroppedPre1970.ShouldBe(6);
// A different tag has its own budget.
(await w.WriteLiveValuesAsync("b", Pre1970(1), CancellationToken.None)).ShouldBeTrue();
log.Warnings.ShouldBe(2);
// Once the interval has passed since the last Warning for "a", it warns again.
advance(TimeSpan.FromSeconds(1));
(await w.WriteLiveValuesAsync("a", Pre1970(1), CancellationToken.None)).ShouldBeTrue();
log.Warnings.ShouldBe(3);
w.DroppedPre1970.ShouldBe(8);
}
[Fact]
public void The_interval_is_ten_minutes() =>
QuestDbHistorianValueWriter.Pre1970WarningInterval.ShouldBe(TimeSpan.FromMinutes(10));
private sealed class ListLogger : ILogger
{
private int _warnings;
public int Warnings => Volatile.Read(ref _warnings);
public IDisposable? BeginScope<TState>(TState state) where TState : notnull => null;
public bool IsEnabled(LogLevel logLevel) => true;
public void Log<TState>(LogLevel logLevel, EventId eventId, TState state, Exception? exception, Func<TState, Exception?, string> formatter)
{
if (logLevel == LogLevel.Warning) Interlocked.Increment(ref _warnings);
}
}
}
tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests/QuestDbSchemaBootstrapperFailureTests.cs
+32
View File
@@ -0,0 +1,32 @@
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests;
public sealed class QuestDbSchemaBootstrapperFailureTests
{
private sealed class ManualClock : TimeProvider
{
private DateTimeOffset _now = new(2026, 10, 11, 0, 0, 0, TimeSpan.Zero);
public override DateTimeOffset GetUtcNow() => _now;
public void Advance(TimeSpan by) => _now += by;
}
[Fact]
public async Task A_failure_is_cached_for_five_seconds_then_retried()
{
var opts = new QuestDbHistorianOptions { ConnectionString = "Host=127.0.0.1;Port=1;Timeout=1" };
var clock = new ManualClock();
var boot = new QuestDbSchemaBootstrapper(new QuestDbConnection(opts), opts, timeProvider: clock);
var first = await Should.ThrowAsync<Exception>(() => boot.EnsureAsync(CancellationToken.None));
// Within the window the SAME failure is re-thrown without another connection attempt.
var second = await Should.ThrowAsync<Exception>(() => boot.EnsureAsync(CancellationToken.None));
second.ShouldBeSameAs(first);
clock.Advance(BootstrapFailureCache.DefaultDuration + TimeSpan.FromSeconds(1));
var third = await Should.ThrowAsync<Exception>(() => boot.EnsureAsync(CancellationToken.None));
third.ShouldNotBeSameAs(first); // the window closed: QuestDB was tried again
}
}
tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests/QuestDbTablesTests.cs
+66
View File
@@ -0,0 +1,66 @@
using Shouldly;
using Xunit;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests;
public sealed class QuestDbTablesTests
{
[Fact]
public void Sample_ddl_without_ttl_is_pinned() =>
QuestDbTables.CreateSample("hist_", null).ShouldBe(
"CREATE TABLE IF NOT EXISTS hist_sample (ts TIMESTAMP, tag SYMBOL, kind BYTE, num_value DOUBLE, " +
"int_value LONG, str_value VARCHAR, status LONG, deleted BOOLEAN, server_ts TIMESTAMP) " +
"TIMESTAMP(ts) PARTITION BY DAY WAL DEDUP UPSERT KEYS(ts, tag)");
[Fact]
public void Sample_ddl_with_ttl_puts_ttl_between_partition_and_wal() =>
QuestDbTables.CreateSample("hist_", 7).ShouldBe(
"CREATE TABLE IF NOT EXISTS hist_sample (ts TIMESTAMP, tag SYMBOL, kind BYTE, num_value DOUBLE, " +
"int_value LONG, str_value VARCHAR, status LONG, deleted BOOLEAN, server_ts TIMESTAMP) " +
"TIMESTAMP(ts) PARTITION BY DAY TTL 7 DAYS WAL DEDUP UPSERT KEYS(ts, tag)");
[Fact]
public void Event_ddl_is_pinned()
{
QuestDbTables.CreateEvent("hist_", null).ShouldBe(
"CREATE TABLE IF NOT EXISTS hist_event (ts TIMESTAMP, event_id VARCHAR, source SYMBOL, alarm_id VARCHAR, " +
"alarm_name VARCHAR, alarm_type_name VARCHAR, severity INT, event_kind VARCHAR, message VARCHAR, " +
"user_name VARCHAR, comment VARCHAR, received_utc TIMESTAMP) " +
"TIMESTAMP(ts) PARTITION BY DAY WAL DEDUP UPSERT KEYS(ts, event_id)");
QuestDbTables.CreateEvent("hist_", 30).ShouldContain("PARTITION BY DAY TTL 30 DAYS WAL DEDUP");
}
[Fact]
public void Tag_ddl_is_pinned_and_never_carries_a_ttl() =>
QuestDbTables.CreateTag("hist_").ShouldBe(
"CREATE TABLE IF NOT EXISTS hist_tag (ts TIMESTAMP, name SYMBOL, data_type BYTE, eng_unit VARCHAR, " +
"description VARCHAR, updated_utc TIMESTAMP) TIMESTAMP(ts) PARTITION BY YEAR WAL DEDUP UPSERT KEYS(ts, name)");
[Fact]
public void The_prefix_is_applied_to_every_table_name()
{
QuestDbTables.All("t1_").ShouldBe(["t1_sample", "t1_event", "t1_tag"]);
QuestDbTables.CreateAll("t1_", 3).ShouldAllBe(s => s.Contains("t1_"));
}
[Fact]
public void An_invalid_prefix_is_refused() =>
Should.Throw<ArgumentException>(() => QuestDbTables.CreateSample("a b", null));
[Fact]
public void Alter_ttl_is_pinned_and_null_removes_it()
{
QuestDbTables.AlterTtl("hist_sample", 30).ShouldBe("ALTER TABLE hist_sample SET TTL 30 DAYS");
QuestDbTables.AlterTtl("hist_sample", null).ShouldBe("ALTER TABLE hist_sample SET TTL 0 DAYS");
}
[Theory]
[InlineData(1, "WEEK", 168)] // 7 days reads back as 1 WEEK
[InlineData(2, "WEEK", 336)]
[InlineData(30, "DAY", 720)]
[InlineData(0, "HOUR", 0)] // no TTL
[InlineData(5, "hour", 5)]
[InlineData(1, "MONTH", -1)] // unknown unit never equals a wanted value
public void Ttl_is_compared_in_hours(long value, string unit, long hours) =>
QuestDbTables.TtlHours(value, unit).ShouldBe(hours);
}
tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests/QuestDbTimeClustersTests.cs
+62
View File
@@ -0,0 +1,62 @@
using Shouldly;
using Xunit;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests;
/// <summary>
/// The shared existence-query clustering (moved here, names unchanged, from the writer's <c>WriteTests</c> and the
/// updater's <c>UpdateTests</c> when the two private copies were consolidated into <see cref="QuestDbTimeClusters"/>).
/// </summary>
public sealed class QuestDbTimeClustersTests
{
private static readonly DateTime T0 = new(2026, 3, 1, 12, 0, 0, DateTimeKind.Utc);
[Fact]
public void Cluster_splits_on_a_gap_over_the_limit_and_at_the_per_cluster_cap()
{
var g = TimeSpan.FromMinutes(1);
QuestDbTimeClusters.Cluster([], g, 10).ShouldBeEmpty();
QuestDbTimeClusters.Cluster([T0], g, 10).ShouldBe([(T0, T0)]);
// exactly the gap stays together; one tick over splits
QuestDbTimeClusters.Cluster([T0, T0 + g], g, 10).ShouldBe([(T0, T0 + g)]);
QuestDbTimeClusters.Cluster([T0, T0 + g + TimeSpan.FromTicks(10)], g, 10)
.ShouldBe([(T0, T0), (T0 + g + TimeSpan.FromTicks(10), T0 + g + TimeSpan.FromTicks(10))]);
// outlier gets its own range
var far = T0.AddYears(1);
QuestDbTimeClusters.Cluster([T0, T0.AddSeconds(1), far], g, 10).ShouldBe([(T0, T0.AddSeconds(1)), (far, far)]);
// cap: 5 close times, max 2 per cluster
var five = Enumerable.Range(0, 5).Select(i => T0.AddSeconds(i)).ToList();
QuestDbTimeClusters.Cluster(five, g, 2)
.ShouldBe([(five[0], five[1]), (five[2], five[3]), (five[4], five[4])]);
}
[Fact]
public void Cluster_splits_where_the_gap_exceeds_the_limit()
{
var gap = QuestDbTimeClusters.ExistenceGap;
var c = QuestDbTimeClusters.Cluster(
[T0.AddHours(2), T0, T0.AddSeconds(30), T0.AddHours(1)], gap);
c.ShouldBe([(T0, T0.AddSeconds(30)), (T0.AddHours(1), T0.AddHours(1)), (T0.AddHours(2), T0.AddHours(2))]);
QuestDbTimeClusters.Cluster([], gap).ShouldBeEmpty();
}
[Fact]
public void The_existence_gap_is_one_minute_and_the_default_cap_is_unbounded()
{
QuestDbTimeClusters.ExistenceGap.ShouldBe(TimeSpan.FromMinutes(1));
var many = Enumerable.Range(0, 5000).Select(i => T0.AddMilliseconds(i)).ToList();
QuestDbTimeClusters.Cluster(many, QuestDbTimeClusters.ExistenceGap).ShouldBe([(many[0], many[^1])]);
}
[Fact]
public void Unsorted_input_is_sorted_before_capping()
{
var times = new[] { T0.AddSeconds(3), T0, T0.AddSeconds(2), T0.AddSeconds(1) };
QuestDbTimeClusters.Cluster(times, TimeSpan.FromMinutes(1), 2)
.ShouldBe([(T0, T0.AddSeconds(1)), (T0.AddSeconds(2), T0.AddSeconds(3))]);
}
[Fact]
public void A_cap_below_one_is_rejected() =>
Should.Throw<ArgumentOutOfRangeException>(() => QuestDbTimeClusters.Cluster([T0], TimeSpan.FromMinutes(1), 0));
}
tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests/QuestDbWalVisibilityTests.cs
+33
View File
@@ -0,0 +1,33 @@
using Shouldly;
using Xunit;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests;
public sealed class QuestDbWalVisibilityTests
{
[Fact]
public async Task Timeout_zero_skips_the_wait_without_touching_the_database()
{
// Port 1 refuses connections: any attempt to query would throw, so returning true proves nothing was opened.
var conn = new QuestDbConnection(new QuestDbHistorianOptions { ConnectionString = "Host=127.0.0.1;Port=1;Timeout=1" });
var wal = new QuestDbWalVisibility(conn, TimeSpan.Zero);
wal.Disabled.ShouldBeTrue();
(await wal.WaitAsync("hist_sample", CancellationToken.None)).ShouldBeTrue();
}
[Fact]
public async Task A_positive_timeout_does_touch_the_database_control()
{
var conn = new QuestDbConnection(new QuestDbHistorianOptions { ConnectionString = "Host=127.0.0.1;Port=1;Timeout=1" });
var wal = new QuestDbWalVisibility(conn, TimeSpan.FromSeconds(1));
wal.Disabled.ShouldBeFalse();
await Should.ThrowAsync<Exception>(() => wal.WaitAsync("hist_sample", CancellationToken.None));
}
[Theory]
[InlineData(5, 5, true)]
[InlineData(6, 5, true)]
[InlineData(4, 5, false)]
public void Reached_is_writerTxn_at_or_beyond_the_target(long writer, long target, bool expected) =>
QuestDbWalVisibility.Reached(writer, target).ShouldBe(expected);
}
tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests.csproj
+27
View File
@@ -0,0 +1,27 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>net10.0</TargetFramework>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
<IsPackable>false</IsPackable>
<IsTestProject>true</IsTestProject>
<RootNamespace>ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests</RootNamespace>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="xunit.v3"/>
<PackageReference Include="Shouldly"/>
<PackageReference Include="Microsoft.NET.Test.Sdk"/>
<PackageReference Include="xunit.runner.visualstudio">
<PrivateAssets>all</PrivateAssets>
<IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
</PackageReference>
</ItemGroup>
<ItemGroup>
<ProjectReference Include="..\..\..\src\Drivers\ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb\ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.csproj"/>
<ProjectReference Include="..\..\..\src\Drivers\ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common\ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.csproj"/>
</ItemGroup>
</Project>

Some files were not shown because too many files have changed in this diff Show More