From 9998d816f1e40c44db5e702d604bb6c47f087124 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Sat, 10 Oct 2026 08:46:48 -0400 Subject: [PATCH 01/23] docs(historian): QuestDB historian backend design (#666) Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe --- .../2026-10-10-questdb-historian-design.md | 250 ++++++++++++++++++ 1 file changed, 250 insertions(+) create mode 100644 docs/plans/2026-10-10-questdb-historian-design.md diff --git a/docs/plans/2026-10-10-questdb-historian-design.md b/docs/plans/2026-10-10-questdb-historian-design.md new file mode 100644 index 00000000..c01068be --- /dev/null +++ b/docs/plans/2026-10-10-questdb-historian-design.md @@ -0,0 +1,250 @@ +# QuestDB historian backend — design + +**Status:** approved section by section on 2026-10-10. Gitea #666. +**Builds on:** `docs/plans/2026-10-09-pluggable-historian-design.md`, which defines the provider seam, typed values, the +HistoryUpdate service and the SQL Server backend, all merged to master as `8ec7a83c`. + +## 0. Decisions (user, 2026-10-10) + +| Topic | Decision | +|---|---| +| HistoryUpdate deletes | **Soft delete.** Rows carry a `deleted` flag. A delete writes a tombstone at the same `(tag, ts)`. Every read and aggregate excludes tombstones. All five HistoryUpdate operations are supported. | +| Write path | **Postgres wire protocol only, via `Npgsql`.** It is already centrally pinned at 10.0.3. No ILP, no new NuGet package. | +| Retention | **Included.** `RetentionDays` is applied as table TTL. QuestDB drops expired day partitions itself. | +| Approach | **B: shared dialect-free core plus per-backend SQL.** The pure logic moves out of the SQL Server backend into `Driver.Historian.Sql.Common`. One conformance suite runs against both backends. | +| Timestamps | **Microseconds.** Postgres-wire timestamps are microsecond resolution, and real source timestamps are at best milliseconds. Two samples less than 1 µs apart within a batch collapse to one, and the first wins. This is documented. | + +Out of scope: +- event HistoryUpdate; +- Part 11 modified values (#668); +- time-weighted aggregates (#671); +- ILP ingest; +- QuestDB authentication beyond the connection string; +- any change to the Gateway backend. + +## 1. Components + +### 1.1 `ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common` (new, no I/O) + +This is extracted from `Driver.Historian.SqlServer` with **no behaviour change**: + +- **`StoredSample`.** It holds the time, the value, the stored status and a deleted flag. It classifies a sample as + usable, good, bad or uncertain. It applies the quality-contract clamp: a NULL value carrying a non-Bad status is + served as `BadNoData`. It also defines the info-bit masks (`CodeBitsMask`, `SeverityBadBit`, `SeverityMask`). +- **`AtTimeEvaluator`.** This is the former `SqlServerHistorianDataSource.AtTimeValue`. It covers interpolation and + stepping, the outage rule (previous neighbour Bad ⇒ null plus that code; previous usable and next Bad ⇒ hold the + previous value as `UncertainDataSubNormal`), stepping forward after the last sample, `T > now` ⇒ Uncertain, and + masking `& ~0x3` before the info bits are set. +- **`BucketRow` + `AggregateEvaluator`.** This is the former `Evaluate`. Each backend's SQL produces the **same + per-bucket statistics**: total count, usable/good/bad/uncertain counts, numeric sum/min/max, first and last value + and time, the time of the min and of the max, and so on. The shared evaluator maps a `BucketRow` to the OPC UA + value and status for each of the 13 supported aggregates. Unsupported aggregates throw + `HistoryAggregateNotSupportedException`. +- **`BucketPlan`.** It handles window normalisation (a reversed window is swapped and returned descending, as built + today), interval 0 ⇒ one bucket, the 100,000-bucket cap, and tick arithmetic. It exposes the bucket start and width + in **ticks** and in **microseconds**. +- **`HistorianColumnMapping`.** It maps a `HistorianValueKind` to the stored data-type code (the former + `SqlHistorianDataTypes.FromKind`). It also maps a `HistorianValue` to its value columns (num/int/str) and back. +- **`HistoryUpdateResults`.** These are the per-item result rules for Insert, Replace, Update, DeleteAtTime and + DeleteRaw given "did a live row exist". They include the NaN/Inf rule (rejected `BadInvalidArgument` on update, + stored as `BadOutOfRange` Empty on the live write) and the rule rejecting an Empty value with a non-Bad status. +- **Shared constants.** The named status consts that both backends emit (`BadOutOfRange` and others) move into one + parity-visible `*StatusCodes` class here. Backend-specific codes stay with their backend. + +`Driver.Historian.SqlServer` then keeps only its T-SQL, the connection, the bootstrapper, the tag catalog and the +seam classes, all calling the shared core. + +### 1.2 `ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb` (new) + +| Class | Responsibility | +|---|---| +| `QuestDbHistorianOptions` | Binds `ServerHistorian:QuestDb` (§4) and validates it. | +| `QuestDbConnection` | Npgsql `NpgsqlDataSource` (pooled), command timeouts, transient-error classification, UTC timestamp reads. | +| `QuestDbSchemaBootstrapper` | Idempotent `CREATE TABLE IF NOT EXISTS` for the three tables, then applies TTL. It is concurrency-safe across two nodes starting together, and caches a failure for 5 s. | +| `QuestDbHistorianValueWriter` | `IHistorianValueWriter` for live samples (§2.2). | +| `QuestDbTagProvisioner` | `IHistorianProvisioning`: upserts `hist_tag` rows. | +| `QuestDbAlarmHistorianWriter` | `IAlarmHistorianWriter`: writes events with the deterministic SHA-256 `event_id` and the same classify rules as SQL Server (unknown ⇒ RetryPlease). | +| `QuestDbHistorianDataSource` | `IHistorianDataSource` (§3), with QuestDB SQL for raw, at-time, events and bucket statistics. | +| `QuestDbHistorianDataUpdater` | `IHistorianDataUpdater` (§2.3), the soft-delete updater. | +| `QuestDbHistorianProvider` | `IHistorianBackendProvider`: `Name = "QuestDb"`, `SupportedKinds` = all six (FrozenSet), one shared connection and bootstrapper. | + +Host changes: +- `HistorianBackendRegistry` gains `"QuestDb"`. +- `ServerHistorianOptionsValidator` validates the QuestDb section, gated on consumers, the same way it gates + SqlServer. +- `Host.csproj` gets a ProjectReference. + +## 2. Schema and writes + +### 2.1 Tables + +QuestDB has no schemas, so the tables carry a configurable `TablePrefix` (default `hist_`, `[A-Za-z0-9_]+`). All +three are **WAL tables with `DEDUP UPSERT KEYS`**, so re-sending a row with the same key replaces it. + +```sql +CREATE TABLE IF NOT EXISTS hist_sample ( + ts TIMESTAMP, tag SYMBOL, kind BYTE, num DOUBLE, int LONG, str VARCHAR, + status LONG, deleted BOOLEAN, server_ts TIMESTAMP +) TIMESTAMP(ts) PARTITION BY DAY WAL DEDUP UPSERT KEYS(ts, tag) [TTL n DAYS]; + +CREATE TABLE IF NOT EXISTS hist_event ( + ts TIMESTAMP, event_id VARCHAR, source SYMBOL, + -- plus the remaining hist.Event columns one-for-one (docs/historian/sqlserver/001_initial.sql), QuestDB-typed +) TIMESTAMP(ts) PARTITION BY DAY WAL DEDUP UPSERT KEYS(ts, event_id) [TTL n DAYS]; + +CREATE TABLE IF NOT EXISTS hist_tag ( + ts TIMESTAMP, name SYMBOL, data_type BYTE, eng_unit VARCHAR, description VARCHAR, updated_utc TIMESTAMP +) TIMESTAMP(ts) PARTITION BY YEAR WAL DEDUP UPSERT KEYS(ts, name); -- ts is always 1970-01-01, which makes this an upsert by name +``` + +Column details: +- `status` holds the uint status code as a LONG. +- `int` also holds Booleans as 0/1, matching SQL Server. +- `kind` is `HistorianValueKind`. +- The exact column types and the TTL syntax are confirmed by T0 against the pinned image. T0 amends this section + with what it finds. + +### 2.2 Live writes (recorder) — insert only if absent, the same as SQL Server + +1. Group the batch by tag and remove duplicates within the batch. **The first value wins.** A timestamp is rounded + to microseconds before this step, so microsecond collisions also collapse. +2. For each tag, fetch the timestamps that already exist in the batch's `[min ts, max ts]` range. **Tombstones are + included**, so a re-sent outbox batch can never resurrect a deleted row or overwrite a HistoryUpdate change. +3. Insert only the new rows, in one parameterised multi-row `INSERT` per tag group. +4. Stored-value rules are shared with SQL Server: + - NaN/±Inf, or any Empty with a non-Bad status, is stored Empty with `BadOutOfRange` or `BadNoData`. + - Comms-loss outage markers are stored as given. + - A tag name over 400 characters is a per-tag failure. +5. Any SQL or transport failure makes the write return `false`, so the recorder retries and the #538 per-tag + quarantine still applies. It never throws. + +The recorder is Primary-only (#537), so step 2 does not race another live writer. The recorder does not wait for +WAL visibility. + +### 2.3 HistoryUpdate (soft delete) + +Every operation is an `INSERT`, and the dedup upsert performs the replace. QuestDB `UPDATE` and `DELETE` are never +issued. "Exists" means a row that is present **and not deleted**. + +| Operation | Effect | Result | +|---|---|---| +| Insert | Write the row when none exists; a tombstone counts as absent | `GoodEntryInserted` / `BadEntryExists` | +| Replace | Overwrite the row when one exists | `GoodEntryReplaced` / `BadNoEntryExists` | +| Update | Write the row either way | `GoodEntryReplaced` or `GoodEntryInserted` | +| DeleteAtTime | Write a tombstone (`deleted = true`, value NULL) where a row exists | `Good` / `BadNoEntryExists` | +| DeleteRaw | Write a tombstone for every live row in the range | the same codes SQL Server returns for the same cases | + +Further rules: +- The per-item rules come from `HistoryUpdateResults`, shared with SQL Server, so the codes are identical. +- Item order is preserved, with exactly one code per item. +- **Read-your-writes.** After its inserts, the updater polls `wal_tables()` until the table's applied transaction + reaches the sequencer transaction it produced, up to `VisibilityTimeout`. On a timeout the write is kept, the codes + are reported as normal, and a Warning is logged. +- The existence checks for Insert and Replace read committed data. If a live recorder sample at the *same* + timestamp is still in flight in the WAL, it can race the updater. Updates normally target past timestamps, so this + is accepted and documented. + +### 2.4 Retention + +When `RetentionDays` is set, the bootstrapper applies TTL to `hist_sample` and `hist_event`, and re-applies it when +the configured value changes. QuestDB then drops expired day partitions, tombstones included. When unset, data is +kept forever. `hist_tag` never expires. + +## 3. Reads + +Every query filters `NOT deleted`. Every value goes through `StoredSample`, so the quality clamp is the same as SQL +Server's. + +- **Raw.** A range on `ts`, `ORDER BY ts [DESC]`, `LIMIT n+1` to detect another page. The dedup key forbids + same-instant ties, so the continuation is just the last timestamp. Bounds follow the existing data-source contract. +- **AtTime.** Find the previous-or-equal neighbour and the next neighbour of each requested time. The default is + `LIMIT 1` seeks on `ts`, batched into chunked statements; T0 picks `ASOF JOIN` for the previous neighbour if it + measures better. `AtTimeEvaluator` then decides the result. +- **Events.** Filtered by source and `ts` range, ordered ascending (reversed windows are swapped, as built today), + and limited to `maxEvents`. +- **Processed.** **One query per node.** + - Bucket index: `(cast(ts AS LONG) - @startMicros) / @intervalMicros`, using microsecond values from `BucketPlan`, + grouped by the index. + - The query returns the full `BucketRow` statistics. Usable, good, bad and uncertain counts come from + `sum(CASE …)` over the status column. + - It does **not** use `SAMPLE BY`, whose alignment rules differ from ours. Computing the index ourselves keeps the + bucket edges identical to SQL Server: start inclusive, end exclusive, a partial final bucket, interval 0, and the + 100k cap. + - The time of the min or max value needs an arg-min style lookup. It is computed with a window function, or with a + second statement if QuestDB lacks one; T0 decides. `AggregateEvaluator` then produces the values. +- **Errors.** A failed read fails only that node, under the same data-source contract as SQL Server. + +## 4. Configuration (`ServerHistorian:QuestDb`) + +| Key | Default | Validation | +|---|---|---| +| `ConnectionString` | — | Required when `Backend=QuestDb` and a consumer is enabled. An Npgsql string, e.g. `Host=questdb;Port=8812;Username=admin;Password=quest;Database=qdb;ServerCompatibilityMode=NoTypeLoading` (T0 confirms the exact keywords). | +| `TablePrefix` | `hist_` | `^[A-Za-z0-9_]+$`. Only checked when a consumer is enabled. | +| `RetentionDays` | unset | ≥ 1 when set | +| `CommandTimeout` | `00:00:30` | > 0 | +| `AutoCreateSchema` | `true` | — | +| `VisibilityTimeout` | `00:00:05` | ≥ 0. 0 disables the wait. | + +The Gateway keys (`Endpoint`, `ApiKey`, `UseTls`) are not required for QuestDb, the same as for SqlServer. + +## 5. Testing + +- **T1 refactor gate.** The SQL Server integration suite (118) and unit suite pass unchanged before and after the + extraction. Each moved piece of logic is falsified once against its existing test. +- **Conformance suite** (`tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Conformance`, a class library of abstract + xUnit tests). Each backend's integration project subclasses it and supplies a fixture and an `IConformanceSeeder`, + which writes rows directly in that backend's schema. + - It covers raw forward, reverse and paging; AtTime, including the outage, after-last and future rules; all 13 + aggregates with the existing hand-computed literals; the quality contract; insert-if-absent writes and + redelivery; the HistoryUpdate result codes; and over-long tag names. + - Each legitimate backend difference is a named, documented override, never a silent skip. Microsecond rounding is + the only expected one. +- **QuestDB-only tests:** + - a tombstone hides the row and a redelivery does not resurrect it; + - the read-your-writes wait; + - TTL is applied, verified through `tables()` metadata; + - concurrent bootstrap from two instances; + - intra-batch microsecond collisions; + - the `hist_tag` upsert; + - transient-failure behaviour, writing against a stopped port. +- **Fixture.** `HISTORIAN_QUESTDB_TEST_CONNECTION`; when it is absent, the tests skip. Each test class uses a unique + throwaway `TablePrefix` (`t__`) and drops its tables on dispose. +- **Host.** Registry and validator tests for `Backend=QuestDb`, beside the SqlServer ones. + +## 6. Environment + +- docker-dev gains a `questdb` service, using an exact image tag that T0 pins (at least two weeks old). It sits behind + a compose profile (`questdb`) so the default rig is unchanged. +- It exposes PG wire `8812` and the web console `9000` on host ports that T0 chooses so they don't collide with + AdminUI `:9000`/`:9200`. +- It uses a named volume. +- The central nodes gain `ServerHistorian__QuestDb__ConnectionString` (`OTOPCUA_HISTORIAN_QUESTDB`), defaulting to the + `questdb` service. +- No new NuGet packages. + +## 7. Live gate (T9, run by the controller) + +On docker-dev, with `Backend=QuestDb`: +1. Only the Primary records, with real kinds and status, one row per value. +2. Raw and Processed HistoryRead return identical results through the raw and UNS NodeIds, and match a direct + QuestDB query. +3. HistoryUpdate covers insert, duplicate, replace, replace-missing and delete-at-time, each confirmed by a read and + by a direct query. After a delete, an outbox redelivery does not bring the row back. A WriteOperate-only key and an + anonymous session are both denied. +4. On failover, recording continues at 1×. +5. Modbus device loss stores a `BadCommunicationError` Empty sample, then the `BadNoCommunication` outage marker, + then Good on recovery. +6. TTL is visible in table metadata when `RetentionDays` is set. + +Evidence goes in `docs/plans/2026-10-10-questdb-historian-live-gate.md`, and the rig is restored afterwards. + +## 8. Work order + +| Wave | Task | Model | +|---|---|---| +| W0 | **T0 spike.** Pin the image, add the compose service, and prove every QuestDB assumption here: Npgsql parameterised multi-row inserts, `DEDUP UPSERT`, `wal_tables()` visibility, TTL syntax, microsecond timestamps over Npgsql, the min/max-time query form, AtTime neighbour lookup, concurrent `CREATE TABLE IF NOT EXISTS`. **Amend §2–§4 with the findings before W1.** | controller + sonnet | +| W1 | T1: extract `Sql.Common`, leaving SQL Server's behaviour unchanged | opus | +| W2 | T2 conformance suite over SQL Server ‖ T3 QuestDB skeleton (options, connection, bootstrapper, fixture) | sonnet ‖ sonnet | +| W3 | T4 writes, provisioning and alarms ‖ T5 reads and aggregates ‖ T6 updater with soft delete | sonnet ‖ opus ‖ sonnet | +| W4 | T7 provider, registry and validator, plus the conformance suite run against QuestDB | sonnet | +| W5 | T8 docs (Historian.md, Configuration.md, CLAUDE.md, docker-dev README), then a Fable whole-branch review, one fix wave, and T9 live gate | sonnet / fable / opus / controller | From b3031dfea9496eb53a18b31aa47c0c5f242e5f66 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Sat, 10 Oct 2026 08:56:15 -0400 Subject: [PATCH 02/23] docs(historian): QuestDB backend implementation plan (#666) Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe --- .../2026-10-10-questdb-historian-plan.md | 869 ++++++++++++++++++ ...10-10-questdb-historian-plan.md.tasks.json | 96 ++ 2 files changed, 965 insertions(+) create mode 100644 docs/plans/2026-10-10-questdb-historian-plan.md create mode 100644 docs/plans/2026-10-10-questdb-historian-plan.md.tasks.json diff --git a/docs/plans/2026-10-10-questdb-historian-plan.md b/docs/plans/2026-10-10-questdb-historian-plan.md new file mode 100644 index 00000000..7bdf912c --- /dev/null +++ b/docs/plans/2026-10-10-questdb-historian-plan.md @@ -0,0 +1,869 @@ +# QuestDB Historian Backend Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: use superpowers-extended-cc:subagent-driven-development (recommended) or +> superpowers-extended-cc:executing-plans to run this plan task by task. Steps use checkbox (`- [ ]`) syntax. + +**Goal:** add `ServerHistorian:Backend=QuestDb`. It reaches full parity with the SQL Server backend, covering +recording, reads, the 13 aggregates and all five HistoryUpdate operations through soft delete, and adds table-TTL +retention. + +**Architecture:** move the database-independent logic of `Driver.Historian.SqlServer` into a new shared library, +`Driver.Historian.Sql.Common`. Build `Driver.Historian.QuestDb` on top of it, using Npgsql over the Postgres wire +protocol. A shared-source conformance suite runs the same hand-computed expectations against both backends. + +**Tech stack:** .NET 10, Npgsql 10.0.3 (already pinned), QuestDB as a WAL table with `DEDUP UPSERT KEYS`, xUnit v3 + +Shouldly, docker-dev. + +**Spec:** `docs/plans/2026-10-10-questdb-historian-design.md`. Its §0 holds the user decisions. The T0 spike amends +§2–§4 before W1 starts. + +## Global Constraints + +These are binding on every task and every reviewer. + +**Packages and build** +- Add no NuGet packages and change no versions. Never edit `Directory.Packages.props`. Referencing `Npgsql` from the + new QuestDb project is allowed, because it is already centrally pinned. +- Every new `src` csproj sets `TreatWarningsAsErrors=true`, `GenerateDocumentationFile=true` and `NoWarn CS1591`, as + `Driver.Historian.SqlServer.csproj` does. `TargetFramework` and `Nullable` are inherited from + `Directory.Build.props`. +- New `src` project names MUST match `ZB.MOM.WW.OtOpcUa.Driver.*`. `StatusCodeParityTests` scans only + `ZB.MOM.WW.OtOpcUa.Driver.*.dll`. + +**Status codes** +- Every OPC UA status code a class emits is a named `const uint` whose name starts with `Good`, `Uncertain` or `Bad`. + Bit masks use names that do NOT start with those words (`CodeBitsMask`, `SeverityBadBit`). +- Any new Driver assembly must be a ProjectReference of `tests/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions.Tests`, so its + DLL lands in that test's bin and gets scanned. + +**SQL Server backend** +- Its observable behaviour must not change. The SqlServer unit and integration suites stay green, and no test there + is deleted. A test may only *move* into the conformance suite, keeping the same name and the same literals. + +**Quality and write rules** +- **Quality contract:** an Empty (null-value) sample is always served with a Bad status. NaN/±Inf and Empty-with-non-Bad + are stored as `BadOutOfRange` or `BadNoData` on the live write, and rejected with `BadInvalidArgument` by HistoryUpdate. +- **Live writes are insert-if-absent**, and the first duplicate in a batch wins. On QuestDB, an existing *tombstone* + also counts as present, so it blocks the insert and the row is never resurrected. +- **QuestDB writes:** every write is an `INSERT`, and the dedup upsert does the replacing. Never issue QuestDB `UPDATE` + or `DELETE` for sample or event data. `DROP TABLE` is allowed only in test cleanup. +- **Soft delete:** a delete writes a tombstone row (`deleted = true`, value columns NULL) at the same `(ts, tag)`. Every + read and aggregate filters `NOT deleted`. + +**Time and buckets** +- **Timestamps:** QuestDB stores microseconds. The QuestDB writer and updater truncate to microseconds + (`ticks - ticks % 10`) before de-duplicating. Conformance tests compare timestamps through + `IConformanceBackend.Round(DateTime)`. +- **Bucket edges must be identical to SQL Server's.** The QuestDB bucket index is + `(cast(ts AS LONG) * 10 - @startUnixTicks) / @intervalTicks`, using integer ticks relative to the Unix epoch. Never use + `SAMPLE BY`. + +**Infrastructure and tests** +- **docker-dev:** QuestDB is an *overlay file*, `docker-dev/docker-compose.questdb.yml`. The rig uses no compose + `profiles:`. Host ports are PG wire `18812` and web console `19000`. + - These ports are already taken and must not be reused: 14330, 14340, 14350, 4840–4845, 4850, 4851, 4860, 4861, 9200, + 9210, 9220, 8089. +- **Test environment variables:** `HISTORIAN_QUESTDB_TEST_CONNECTION`, alongside the existing + `HISTORIAN_SQL_TEST_CONNECTION`. When a variable is absent, its tests skip through `Assert.Skip`. +- **Regression tests** must be falsified: revert the production change, watch the test go red, restore it, and record + what you observed. +- **Shell:** zsh, so quote globs and avoid a bare `====`. +- **Building:** put two projects in one `dotnet build` and it fails with MSB1008, so build them separately or use the + `.slnx`. + +**User decisions (already made):** +- "Soft delete (Recommended)": the `deleted` flag plus tombstones, with all five HistoryUpdate operations. +- "PG wire only (Recommended)": Npgsql for everything, no ILP. +- "Include TTL (Recommended)": `RetentionDays` becomes table TTL. +- Approach B: a shared dialect-free core plus per-backend SQL, and one conformance suite run against both backends. +- Microsecond timestamps; two samples in one batch landing on the same microsecond collapse to one (first wins). +- Every design section was approved as written (2026-10-10). The plan gets a Fable review, then the session is + prepared for compaction before executing with subagents. + +**Plan rulings (made while planning, from the code inventory):** +- **R1:** the spec §6 compose *profile* becomes an overlay file. The rig has no profiles; each optional fixture is + `docker-compose..yml`. +- **R2:** the conformance suite is a **shared-source directory**, `tests/Drivers/Historian.Conformance/`, which has no + csproj. Each integration project includes it with + ``. The reason is that only `xunit.v3` + is pinned, and a class library referencing it would become a test project. +- **R3:** the shared types become `public` in `Sql.Common`, so no InternalsVisibleTo is needed for the two backends. + They are renamed from their `Sql*` names to neutral ones (table in T1). The parity test keys on field names, not + class names. + +--- + +## Waves, models and parallel safety + +| Wave | Tasks (‖ = parallel) | Models | +|---|---|---| +| W0 | T0 QuestDB spike + overlay compose + spec amendments | **controller** (live docker) | +| W1 | T1 extract `Sql.Common` | **opus** | +| W2 | T2 conformance suite over SqlServer ‖ T3 QuestDB skeleton | sonnet ‖ sonnet | +| W3 | T4 writes ‖ T5 reads + aggregates ‖ T6 updater | sonnet ‖ **opus** ‖ sonnet | +| W4 | T7 provider + Host wiring + QuestDB conformance | sonnet | +| W5 | T8 docs → **fable** whole-branch review → one fix wave (**opus**) + scoped re-review (sonnet) → T9 live gate | sonnet / fable / opus / controller | + +Reviewers: +- opus for T1 and T5; +- sonnet for the rest; +- sonnet for scoped re-reviews. + +Each task runs in its own worktree, `.worktrees/qdb-T`, branched from the current `feat/questdb-historian` head. +Merges happen in ascending task order within a wave, with `dotnet build ZB.MOM.WW.OtOpcUa.slnx` after each merge. + +**File ownership.** A task edits only the files it owns. Anything else needs a controller ruling before it is +touched. + +| Path | Owner | +|---|---| +| `docker-dev/docker-compose.questdb.yml`, `docs/plans/2026-10-10-questdb-spike.md`, spec §2–§4 amendments | T0 | +| `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/**`, `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/**` | T1 | +| `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/**`, `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.Tests/**` | T1 | +| `ZB.MOM.WW.OtOpcUa.slnx`, `tests/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions.Tests/*.csproj` | T1 adds Sql.Common; then T3 adds the QuestDb lines | +| `tests/Drivers/Historian.Conformance/**`, `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/**` | T2 | +| `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/{csproj,QuestDbHistorianOptions.cs,QuestDbConnection.cs,QuestDbSchemaBootstrapper.cs,QuestDbTables.cs}` | T3 | +| `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests/**` (csproj + options/bootstrap unit tests) | T3 | +| `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/{csproj,QuestDbFixture.cs,BootstrapTests.cs}` | T3 | +| QuestDb `QuestDbHistorianValueWriter.cs`, `QuestDbTagProvisioner.cs`, `QuestDbAlarmHistorianWriter.cs`; IT `WriteTests.cs`, `ProvisioningTests.cs`, `AlarmWriterTests.cs` | T4 | +| QuestDb `QuestDbHistorianDataSource.cs`, `QuestDbAggregateQueries.cs`; IT `ReadTests.cs` | T5 | +| QuestDb `QuestDbHistorianDataUpdater.cs`, `QuestDbWalVisibility.cs`; IT `UpdateTests.cs`, `TombstoneTests.cs` | T6 | +| QuestDb `QuestDbHistorianProvider.cs`; IT `QuestDbConformanceBackend.cs`, `Conformance*.cs`; Host `HistorianBackendRegistry.cs`, `ServerHistorianOptionsValidator.cs`, `Host.csproj`; registry and validator tests; `docker-dev/docker-compose.yml` central env | T7 | +| `docs/Historian.md`, `docs/Configuration.md`, `CLAUDE.md`, `docker-dev/README.md`, the spec (final as-built pass) | T8 | +| `docs/plans/2026-10-10-questdb-historian-live-gate.md` | T9 | + +In W3, T4, T5 and T6 must NOT edit T3's files. If one of them needs a connection helper T3 did not provide, it adds a +`private static` helper inside its own file, and the fix wave dedupes. None of T4–T6 edit any csproj or the slnx, which +T3 already wired. + +--- + +### Task 0: QuestDB spike, overlay compose and spec amendments + +**Goal:** prove every QuestDB behaviour the design assumes, against a pinned image, before any code is built on it. +The findings are written into spec §2–§4. + +**Files:** +- Create: `docker-dev/docker-compose.questdb.yml` +- Create: `docs/plans/2026-10-10-questdb-spike.md` +- Modify: `docs/plans/2026-10-10-questdb-historian-design.md` (§2–§4: replace assumptions with confirmed forms, and mark + each one "T0-confirmed") + +**Acceptance Criteria:** +- [ ] The overlay pins an exact `questdb/questdb:` tag. The tag's release date is at least two weeks before + 2026-10-10, and is recorded in the spike doc. +- [ ] The overlay exposes `18812:8812` and `19000:9000`, uses a named volume `otopcua-questdb-data`, and sets the + container name `otopcua-dev-questdb-1`. It is brought up with + `docker compose -f docker-dev/docker-compose.yml -f docker-dev/docker-compose.questdb.yml up -d questdb`. +- [ ] The spike doc records a working Npgsql connection string and answers Q1–Q12 below, each with the exact SQL or + C# that worked and its observed output. +- [ ] Spec §2.1 has the confirmed DDL, §2.3 the confirmed visibility-wait query, §3 the confirmed bucket-statistics and + neighbour query forms, and §4 the confirmed connection-string keywords. + +**Verify:** `docker exec otopcua-dev-questdb-1 true` exits 0. The spike harness output is pasted into the spike doc. + +**Steps:** + +- [ ] **Step 1: Create the overlay.** + +```yaml +# docker-dev/docker-compose.questdb.yml — optional QuestDB fixture for the QuestDB historian backend. +# Up: docker compose -f docker-dev/docker-compose.yml -f docker-dev/docker-compose.questdb.yml up -d questdb +services: + questdb: + image: questdb/questdb: # T0 pins an exact tag ≥ 2 weeks old + container_name: otopcua-dev-questdb-1 + ports: + - "18812:8812" # PG wire + - "19000:9000" # web console / REST + volumes: + - otopcua-questdb-data:/var/lib/questdb + healthcheck: + test: ["CMD-SHELL", "curl -fs http://localhost:9003/status || exit 1"] + interval: 10s + timeout: 5s + retries: 12 +volumes: + otopcua-questdb-data: +``` + +- [ ] **Step 2: Write a throwaway C# harness** in the scratchpad, not in the repo. It is a console project that + references `Npgsql`, restored from the existing central pin via a temporary `Directory.Packages.props` copy, or + run with `dotnet run` inside a scratch folder under the repo with `ManagePackageVersionsCentrally`. Answer each + question with real output: + - **Q1.** The connection-string keywords that work: test `ServerCompatibilityMode=NoTypeLoading` with and without + it. Also the parameter types needed for `TIMESTAMP` (UTC `DateTime`), `BYTE`, `SYMBOL`, `VARCHAR`, `BOOLEAN` and + `LONG`. + - **Q2.** Whether a parameterised multi-row `INSERT INTO t VALUES (…),(…)` works, and the largest row count per + statement that works reliably. Pick the batch size: ≤ 1000 rows, capped by any parameter limit found. + - **Q3.** `CREATE TABLE … WAL DEDUP UPSERT KEYS(ts, tag)`: re-inserting the same key replaces the row. Show + before and after. + - **Q4.** The `wal_tables()` columns. After an insert, how to know the transaction we produced and wait until it is + applied (`writerTxn >= sequencerTxn`). Measure the typical lag over 20 runs. + - **Q5.** TTL: the syntax at create (`TTL n DAYS`), and `ALTER TABLE … SET TTL n DAYS`. Where TTL shows up in + `tables()` metadata. + - **Q6.** Timestamp precision through Npgsql: a value with 100 ns ticks (`…1234567`) comes back as `…123456` (truncated) + or `…123457` (rounded)? Confirm that our pre-truncation makes the round trip exact. + - **Q7.** Two connections run `CREATE TABLE IF NOT EXISTS` for the same table at once, 20 times. Record any error + text, and the catch rule that makes it idempotent. + - **Q8.** Bucket statistics: one `GROUP BY` over the expression `(cast(ts AS LONG)*10 - $1) / $2`, with + `sum(CASE WHEN … THEN 1 ELSE 0 END)`, `avg`, `min`, `max`, `sum`. Plus the first and last usable sample per bucket, + with its value and time. Try `first()`/`last()` with a filter, or a window `row_number() OVER (PARTITION BY b + ORDER BY ts)`. Plus arg-min/arg-max time, via a window function or a second statement. + - **Q9.** AtTime neighbour lookup: chunked `LIMIT 1` seeks (`ts <= t ORDER BY ts DESC LIMIT 1` and + `ts > t ORDER BY ts LIMIT 1`, combined with `UNION ALL` per time) versus `ASOF JOIN`. Time both for 100 timestamps + and pick one. + - **Q10.** VARCHAR and SYMBOL accept a 400-character string, and `SYMBOL` capacity is fine for tens of thousands of + tags. + - **Q11.** An existing-timestamp lookup for a batch: `WHERE tag=$1 AND ts >= $2 AND ts <= $3` (a range), or an `IN` + list. Pick the one that works with parameters. + - **Q12.** `DROP TABLE IF EXISTS` works for test cleanup, and `tables()` lists the tables with a given prefix. + +- [ ] **Step 3: Write the spike doc** with the image tag and date, the connection string, and Q1–Q12 (SQL, output and + decision for each). +- [ ] **Step 4: Amend the spec §2–§4** in place with the confirmed forms. If any assumption failed, write the + replacement design inline and add a ledger ruling. If the failure breaks a user decision, stop and ask. +- [ ] **Step 5: Commit** + +```bash +git add docker-dev/docker-compose.questdb.yml docs/plans/2026-10-10-questdb-spike.md docs/plans/2026-10-10-questdb-historian-design.md +git commit -m "spike(historian): QuestDB behaviours confirmed against pinned image; spec §2–§4 amended + +Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe" +``` + +```json:metadata +{"files":["docker-dev/docker-compose.questdb.yml","docs/plans/2026-10-10-questdb-spike.md","docs/plans/2026-10-10-questdb-historian-design.md"],"verifyCommand":"docker exec otopcua-dev-questdb-1 true","acceptanceCriteria":["pinned image ≥2 weeks old","overlay ports 18812/19000","Q1–Q12 answered with output","spec §2–§4 amended"],"modelTier":"frontier","model":"controller","wave":0} +``` + +--- + +### Task 1: Extract `Driver.Historian.Sql.Common` (no behaviour change) + +**Goal:** move every database-independent piece of `Driver.Historian.SqlServer` into a new shared library. The SQL +Server backend then calls it, with byte-identical behaviour. + +**Files:** +- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.csproj` +- Create in that project: `HistorianStatusCodes.cs`, `StatusBits.cs`, `HistorianInfoBits.cs`, `HistorianDataTypes.cs`, + `SampleColumns.cs`, `HistorianSampleRow.cs`, `StoredSample.cs`, `BucketRow.cs`, `AggregateFamily.cs`, + `AggregateEvaluator.cs`, `AtTimeEvaluator.cs`, `BucketPlan.cs`, `LiveWritePreparer.cs`, `HistoryUpdatePlanner.cs`, + `HistorianTagNames.cs`, `BootstrapFailureCache.cs`, `HistorianSchemaMissingException.cs`, `AlarmEventIds.cs` +- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/` (csproj + tests) +- Modify: every file in `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/` that contained the moved code + (`SqlServerHistorianDataSource.cs`, `SqlAggregateQueries.cs`, `SqlHistorianDataTypes.cs` (delete), + `SqlHistorianStatusCodes.cs` (keep only SQL Server-only codes, or delete if none), `SampleRowMapper.cs`, + `SqlServerHistorianValueWriter.cs`, `SqlServerHistorianDataUpdater.cs`, `SqlServerTagCatalog.cs`, + `SqlServerSchemaBootstrapper.cs`, the csproj) +- Modify: the SqlServer unit tests that tested moved code. Move them into `Sql.Common.Tests` under the same test name. +- Modify: `ZB.MOM.WW.OtOpcUa.slnx` (add both new projects alphabetically, next to lines 30 and 104) +- Modify: `tests/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions.Tests/ZB.MOM.WW.OtOpcUa.Core.Abstractions.Tests.csproj` (add a + ProjectReference to Sql.Common) + +**Acceptance Criteria:** +- [ ] `Sql.Common` has no reference to `Microsoft.Data.SqlClient` or any other database client. Its references are + Core.Abstractions, Server/Runtime (only if needed for `HistorizationValue`) and + `Microsoft.Extensions.Logging.Abstractions`. +- [ ] The public API matches the signatures in Step 1 exactly. T2–T7 depend on them. +- [ ] The SqlServer integration suite passes with the same test count as before the change, which is recorded in the + report. The SqlServer unit suite plus the moved tests in `Sql.Common.Tests` has the same total count. +- [ ] New `Sql.Common.Tests` unit tests cover `AtTimeEvaluator`, `AggregateEvaluator`, `BucketPlan`, `LiveWritePreparer` + and `HistoryUpdatePlanner` directly, using the literals from the SqlServer integration tests. Each one is + falsified once. +- [ ] StatusCodeParity passes, and its constant count does not drop. + +**Verify:** +- `dotnet build ZB.MOM.WW.OtOpcUa.slnx` gives 0 errors. +- `dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests` passes. +- `dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.Tests` passes. +- With `HISTORIAN_SQL_TEST_CONNECTION` set, `dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests` + passes with the same count as before. +- `dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions.Tests --filter "FullyQualifiedName~StatusCodeParity"` + passes. + +**Steps:** + +- [ ] **Step 0: Record the baseline.** Run both SqlServer test projects and the parity filter, and write the exact + pass counts into the report. + +- [ ] **Step 1: Create the shared API.** Namespace: `ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common`. Each moved body is + transplanted verbatim from the source named in the comment; only names and visibility change. + +```csharp +// HistorianStatusCodes.cs — from SqlHistorianStatusCodes + SqlHistorianUpdateStatusCodes (all consts, same values) +public static class HistorianStatusCodes +{ + public const uint Good = 0x00000000u, GoodEntryInserted = 0x00A20000u, GoodEntryReplaced = 0x00A30000u; + public const uint BadNoData = 0x809B0000u, BadEntryExists = 0x809F0000u, BadNoEntryExists = 0x80A00000u; + public const uint BadInvalidTimestamp = 0x80230000u, BadAggregateInvalidInputs = 0x80D60000u; + public const uint UncertainDataSubNormal = 0x40A40000u, BadOutOfRange = 0x803C0000u, BadInvalidArgument = 0x80AB0000u; +} +// StatusBits.cs / HistorianInfoBits.cs — moved verbatim from SqlHistorianStatusCodes.cs (made public) +// HistorianDataTypes.cs — from SqlHistorianDataTypes: DefaultForEmpty, byte FromKind(HistorianValueKind) +// SampleColumns.cs — record struct SampleColumns(byte ValueKind, double? NumValue, long? IntValue, string? StrValue) +// + static class SampleColumnMapper { ToColumns(HistorianValue); FromColumns(byte,double?,long?,string?); +// int ToDbStatus(uint); uint FromDbStatus(int); long ToLongStatus(uint) => status; uint FromLongStatus(long) } +// HistorianSampleRow.cs — record struct HistorianSampleRow(DateTime TimeUtc, HistorianValue Value, uint StatusCode) (was SqlSampleRow) +public readonly record struct StoredSample(DateTime TimeUtc, HistorianValue Value, uint Status) +{ + public bool IsUsable { get; } // verbatim rule from SqlServerHistorianDataSource + public bool IsGood { get; } + /// Applies the quality clamp (NULL value + non-Bad ⇒ BadNoData) — from ReadStored line 533. + public static StoredSample FromStored(DateTime timeUtc, HistorianValue value, uint status); +} +public sealed class BucketRow // fields verbatim from the private BucketRow, as public get/set properties +{ + public long Bucket { get; set; } public int Total { get; set; } public int Usable { get; set; } + public int Good { get; set; } public int Bad { get; set; } public int NonNumeric { get; set; } + public double? Avg { get; set; } public double? Min { get; set; } public double? Max { get; set; } public double? Sum { get; set; } + public HistorianValue First { get; set; } public HistorianValue Last { get; set; } + public double? Extreme { get; set; } public DateTime? ExtremeTime { get; set; } +} +public enum AggregateFamily { Simple, StartEnd, ActualTime } +public static class AggregateFamilies { public static AggregateFamily? FamilyOf(HistoryAggregateType a); } // from SqlAggregateQueries.FamilyOf +public static class AggregateEvaluator +{ + public static DataValueSnapshot Evaluate(HistoryAggregateType a, BucketRow r, DateTime bucketStart, DateTime now); // verbatim + public static DataValueSnapshot NoData(DateTime t, DateTime now); // verbatim +} +public static class AtTimeEvaluator +{ + public static DataValueSnapshot Evaluate(DateTime t, StoredSample? prev, StoredSample? next, DateTime serverTime, DateTime now); // verbatim AtTimeValue +} +public sealed class BucketPlan // from ReadProcessedAsync lines 355-370 and 395-408 +{ + public const int MaxBuckets = 100_000; + public DateTime StartUtc { get; } public DateTime EndUtc { get; } public bool Reversed { get; } + public long IntervalTicks { get; } public long BucketCount { get; } + public long StartUnixTicks => (StartUtc - DateTime.UnixEpoch).Ticks; // for QuestDB's (cast(ts AS LONG)*10 - start)/interval + /// BucketCount > MaxBuckets (param name "interval") — same as today. + public static BucketPlan Create(DateTime startUtc, DateTime endUtc, TimeSpan interval); + public DateTime BucketStart(long index); + /// One snapshot per bucket in window order; missing buckets get ; reversed if the window was. + public IReadOnlyList Assemble(IReadOnlyDictionary byBucket, Func empty); +} +public static class LiveWritePreparer // from SqlServerHistorianValueWriter.Prepare, plus a resolution parameter +{ + /// 1 = keep 100 ns (SQL Server); 10 = truncate to µs BEFORE de-duplication (QuestDB). + public static List Prepare(IReadOnlyList values, Func utcNow, long resolutionTicks = 1); +} +public enum UpsertOutcome { Inserted, Replaced } +public sealed class UpdatePlan +{ + public uint?[] Results { get; } // pre-decided codes (invalid ts, NaN, superseded duplicate…); null = decided by the backend + public IReadOnlyList<(int Index, HistorianSampleRow Row)> Winners { get; } // rows to send, de-duplicated per mode +} +public static class HistoryUpdatePlanner // pre/post passes from SqlServerHistorianDataUpdater.UpdateDataAsync lines 56-108 +{ + public static UpdatePlan Plan(string tag, HistoryUpdateMode mode, IReadOnlyList values, long resolutionTicks = 1); + /// ticks of each winner that the backend actually wrote → Inserted/Replaced; absent ⇒ not written. + public static IReadOnlyList Complete(UpdatePlan plan, HistoryUpdateMode mode, IReadOnlyDictionary outcomeByTick); + public static IReadOnlyList DeleteAtTimeResults(IReadOnlyList times, ISet deletedTicks, bool tagKnown); + public static uint DeleteRawResult(bool tagKnown, long rowsAffected); +} +public static class HistorianTagNames { public const int MaxLength = 400; public static bool IsValid(string? name); } // from SqlServerTagCatalog.IsValidName +public sealed class BootstrapFailureCache // the 5 s negative-cache mechanism from SqlServerSchemaBootstrapper +{ + public BootstrapFailureCache(TimeProvider clock, TimeSpan duration); + public void ThrowIfCached(); public void Record(Exception ex); public void Clear(); +} +// HistorianSchemaMissingException.cs — moved from SqlServerHistorianOptions.cs (public sealed, ctor(string message)) +// AlarmEventIds.cs — public static string ComputeEventId(...) moved verbatim from SqlServerAlarmHistorianWriter (same signature) +``` + +- [ ] **Step 2: Point the SqlServer backend at the shared lib.** + - Replace each moved body with a call into `Sql.Common`. + - Delete the now-empty SqlServer classes. + - Keep `SqlAggregateQueries`' SQL text and `ReadBucket`, now filling the shared `BucketRow`. + - Keep `SampleRowMapper`'s TVP builders, which take `HistorianSampleRow`. +- [ ] **Step 3: Move the SqlServer unit tests for the moved code** into `Sql.Common.Tests`, keeping test names and + literals: `StatusCodeOracleTests`, `ValueWriterPrepareTests`, the column round-trip parts of + `SampleRowMapperTests`, and the data-type table test. The TVP parts stay in SqlServer.Tests. +- [ ] **Step 4: Add direct unit tests that did not exist before.** For example: + +```csharp +[Fact] public void AtTime_inside_an_outage_returns_the_bad_neighbours_code_with_no_value() +{ + var t0 = new DateTime(2026,10,9,8,0,0,DateTimeKind.Utc); + var prev = new StoredSample(t0, HistorianValue.Empty, 0x80310000u); + var next = new StoredSample(t0.AddSeconds(10), HistorianValue.FromDouble(5), 0); + var r = AtTimeEvaluator.Evaluate(t0.AddSeconds(5), prev, next, t0, t0.AddHours(1)); + r.StatusCode.ShouldBe(0x80310000u); r.Value.ShouldBeNull(); +} +[Fact] public void BucketPlan_rejects_more_than_100k_buckets() + => Should.Throw(() => + BucketPlan.Create(DateTime.UnixEpoch, DateTime.UnixEpoch.AddSeconds(100_001), TimeSpan.FromSeconds(1))); +[Fact] public void Prepare_truncates_to_microseconds_before_dedup_when_resolution_is_10() +{ + var t = new DateTime(2026,10,9,8,0,0,DateTimeKind.Utc).AddTicks(1234567); + var rows = LiveWritePreparer.Prepare([new(t, HistorianValue.FromDouble(1), 0), new(t.AddTicks(3), HistorianValue.FromDouble(2), 0)], () => t, 10); + rows.Count.ShouldBe(1); rows[0].TimeUtc.Ticks.ShouldBe(t.Ticks - t.Ticks % 10); rows[0].Value.Number.ShouldBe(1); +} +``` + + Add the matching `AggregateEvaluator` cases, using the 8-row ProcessedRead literals from + `ProcessedReadTests.Numeric_aggregates_over_the_fixture`. Build the `BucketRow`s by hand from tag A's samples. +- [ ] **Step 5: Wire the solution and parity test.** Add both projects to the slnx, and the ProjectReference to the + Core.Abstractions.Tests csproj. +- [ ] **Step 6: Run the Verify commands.** Counts must equal the Step 0 baseline, plus the new direct tests. Falsify + each new direct test once by reverting the rule it pins. +- [ ] **Step 7: Commit** with `refactor(historian): extract Driver.Historian.Sql.Common from the SQL Server backend (no + behaviour change)` and the Claude-Session trailer. + +```json:metadata +{"files":["src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/","tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/","src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/","tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.Tests/","ZB.MOM.WW.OtOpcUa.slnx","tests/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions.Tests/ZB.MOM.WW.OtOpcUa.Core.Abstractions.Tests.csproj"],"verifyCommand":"dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests (env set) — same count as baseline","acceptanceCriteria":["no DB client ref in Sql.Common","public API exactly as Step 1","SqlServer suites same counts","direct evaluator tests falsified","parity green"],"modelTier":"frontier","model":"opus","wave":1} +``` + +--- + +### Task 2: Conformance suite (shared source), run against SQL Server + +**Goal:** one backend-neutral suite of hand-computed behaviour tests, compiled into the SqlServer integration project +and passing there. T7 later compiles the same suite into the QuestDB project. + +**Files:** +- Create: `tests/Drivers/Historian.Conformance/IConformanceBackend.cs`, `ConformanceData.cs`, + `RawReadConformance.cs`, `AtTimeConformance.cs`, `ProcessedConformance.cs`, `QualityContractConformance.cs`, + `LiveWriteConformance.cs`, `UpdateConformance.cs`, `TagNameConformance.cs` +- Modify: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/*.csproj` (add + `` and a ProjectReference to Sql.Common) +- Create in the SqlServer IT project: `SqlServerConformanceBackend.cs`, `SqlServerConformance.cs` (one concrete + subclass per conformance class) +- Modify in the SqlServer IT project: remove from the original files the generic tests that moved into the suite + (`RawReadTests`, `AtTimeReadTests`, `ProcessedReadTests`, `QualityContractTests`, and the generic parts of + `WriteTests`, `UpdateTests` and `LongNameTests`). SQL Server-specific tests stay where they are. + +**Acceptance Criteria:** +- [ ] Every moved test keeps its method name and literal expectations. `git log -p` shows a move, not a rewrite. +- [ ] The SqlServer IT total count after the change is at least the pre-change count, and the report has the before + and after count. +- [ ] Suite code references only `IConformanceBackend` and Core.Abstractions types. There is no `SqlServer*` type and + no T-SQL in `tests/Drivers/Historian.Conformance`. +- [ ] Timestamp assertions go through `B.Round(t)`. +- [ ] Any test that cannot hold for every backend is a `virtual` member with a doc comment naming the difference. + None should be needed for SQL Server. + +**Verify:** with `HISTORIAN_SQL_TEST_CONNECTION` set, `dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests` +passes with a count of at least the baseline. Without the variable, every test skips. + +**Steps:** + +- [ ] **Step 1: Define the backend contract.** + +```csharp +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Conformance; +public interface IConformanceBackend : IAsyncDisposable +{ + /// 1 for SQL Server (100 ns), 10 for QuestDB (µs). + long ResolutionTicks { get; } + DateTime Round(DateTime t) => new(t.Ticks - t.Ticks % ResolutionTicks, DateTimeKind.Utc); + IHistorianDataSource CreateDataSource(TimeProvider? clock = null); + IHistorianValueWriter CreateValueWriter(Func? utcNow = null); + IHistorianDataUpdater CreateUpdater(); + /// Writes rows directly in the backend's schema (never through the writer), status verbatim. + Task SeedAsync(string tag, params (DateTime T, HistorianValue V, uint Status)[] samples); + /// Live (non-deleted) rows for a tag, ascending, read directly from storage. + Task> ReadBackAsync(string tag); +} +public abstract class ConformanceBase : IAsyncLifetime +{ + protected IConformanceBackend B { get; private set; } = null!; + protected abstract Task CreateBackendAsync(); // calls fixture.SkipIfUnavailable() + public async ValueTask InitializeAsync() => B = await CreateBackendAsync(); + public async ValueTask DisposeAsync() => await B.DisposeAsync(); +} +``` + + `ConformanceData` holds `T0`, `At(seconds)`, the status consts and `SeedAAsync(B)`, verbatim from `ReadTestDb`. +- [ ] **Step 2: Move the generic tests** listed in the T1 inventory (Raw, AtTime, Processed literals, the quality + contract, live-write redelivery and first-wins, update result codes, tag names over 400 characters) into the + suite classes. Write each one as an abstract class `RawReadConformance : ConformanceBase` with `[Fact]` methods. +- [ ] **Step 3: Implement `SqlServerConformanceBackend`.** It reuses `SqlHistorianFixture`, a unique schema and the + existing seeding SQL from `ReadTestDb`. Add one concrete class per suite: + `[Collection(SqlHistorianCollection.Name)] public sealed class SqlServerRawReadConformance(SqlHistorianFixture fx) : RawReadConformance`. +- [ ] **Step 4: Run the tests and compare counts.** Commit with + `test(historian): backend conformance suite (shared source), run against SQL Server` and the trailer. + +```json:metadata +{"files":["tests/Drivers/Historian.Conformance/","tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/"],"verifyCommand":"dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests (env set) ≥ baseline count","acceptanceCriteria":["tests moved by name with same literals","count ≥ baseline","suite is backend-neutral","Round() for timestamps"],"modelTier":"standard","model":"sonnet","wave":2} +``` + +--- + +### Task 3: QuestDB skeleton (options, connection, bootstrapper, fixture) + +**Goal:** a building `Driver.Historian.QuestDb` project. It has validated options, a pooled Npgsql connection, an +idempotent and concurrency-safe schema bootstrapper with TTL, and an env-gated integration fixture. + +**Files:** +- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.csproj` + (references Core.Abstractions, Core.AlarmHistorian, Server/Runtime and Sql.Common; packages `Npgsql` and + `Microsoft.Extensions.Logging.Abstractions`; InternalsVisibleTo the two QuestDb test assemblies) +- Create: `QuestDbHistorianOptions.cs`, `QuestDbConnection.cs`, `QuestDbTables.cs`, `QuestDbSchemaBootstrapper.cs` +- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests/` (csproj, `QuestDbHistorianOptionsTests.cs`, + `QuestDbTablesTests.cs`) +- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/` (csproj with the + conformance-include line and a ProjectReference to Sql.Common, `QuestDbFixture.cs`, `BootstrapTests.cs`) +- Modify: `ZB.MOM.WW.OtOpcUa.slnx`, and the Core.Abstractions.Tests csproj (ProjectReference to the QuestDb project) + +**Acceptance Criteria:** +- [ ] `QuestDbHistorianOptions` has `ConnectionString` (default `""`), `TablePrefix` (`"hist_"`, regex + `^[A-Za-z0-9_]+$`), `RetentionDays` (`int?`, ≥ 1), `CommandTimeout` (30 s, > 0), `AutoCreateSchema` (`true`) and + `VisibilityTimeout` (5 s, ≥ 0). `Validate()` returns error strings that name the key + `ServerHistorian:QuestDb:`. +- [ ] `QuestDbTables` produces the three `CREATE TABLE IF NOT EXISTS` statements, as confirmed by T0, for a given prefix + and retention. Unit tests pin the exact text. +- [ ] The bootstrapper is idempotent and survives two concurrent instances, using the catch rule from T0 Q7. It + re-applies a changed TTL. It uses `BootstrapFailureCache` for the 5 s negative cache. +- [ ] `QuestDbFixture` reads `HISTORIAN_QUESTDB_TEST_CONNECTION` and skips when it is absent. `UniquePrefix()` returns + `"t" + 12 hex + "_"`. `DropPrefixAsync(prefix)` drops every table with that prefix, found through `tables()`. +- [ ] `BootstrapTests` cover: + - tables created; + - a second run is a no-op; + - two concurrent bootstrappers both succeed; + - TTL is visible in metadata when `RetentionDays=7` and absent when unset; + - `AutoCreateSchema=false` against a missing table throws `HistorianSchemaMissingException` from Sql.Common. + +**Verify:** +- `dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests` passes. +- With `HISTORIAN_QUESTDB_TEST_CONNECTION` set, `dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests --filter "FullyQualifiedName~Bootstrap"` + passes. +- Without the variable, every test skips. + +**Steps:** +- [ ] Step 1: write the options with `Validate()` and its unit tests (red, then green). +- [ ] Step 2: write `QuestDbTables` DDL builders (use the spec §2.1 *as amended by T0*), with unit tests pinning the + text for prefix `hist_` with and without TTL. +- [ ] Step 3: write `QuestDbConnection`. + - Members: + - `NpgsqlDataSource` built once; + - `OpenAsync(ct)`; + - `CreateCommand(conn, sql)` with `CommandTimeout`; + - `static bool IsTransient(Exception)`, true for `NpgsqlException` with `IsTransient`, socket and IO errors, and + timeouts; + - `static DateTime ReadUtc(NpgsqlDataReader r, int i)`; + - `string T(string table)`, which returns prefix + table. + - Use the parameter typing from T0 Q1. +- [ ] Step 4: write the bootstrapper and fixture. Bring up the overlay + (`docker compose -f docker-dev/docker-compose.yml -f docker-dev/docker-compose.questdb.yml up -d questdb`), then + write and run `BootstrapTests`. +- [ ] Step 5: wire the slnx and the parity-test reference, then build the solution. +- [ ] Step 6: commit `feat(historian): QuestDB backend skeleton — options, connection, schema bootstrap with TTL` with + the trailer. + +```json:metadata +{"files":["src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/","tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests/","tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/","ZB.MOM.WW.OtOpcUa.slnx"],"verifyCommand":"dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests --filter FullyQualifiedName~Bootstrap (env set)","acceptanceCriteria":["options validate naming keys","DDL pinned by unit tests","bootstrap idempotent + concurrent + TTL","fixture env-gated with unique prefix"],"modelTier":"standard","model":"sonnet","wave":2} +``` + +--- + +### Task 4: QuestDB writes (value writer, tag provisioner, alarm writer) + +**Goal:** the recorder, provisioning and alarm-history seams write to QuestDB with the same semantics as SQL Server. + +**Files:** +- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianValueWriter.cs`, + `QuestDbTagProvisioner.cs`, `QuestDbAlarmHistorianWriter.cs` +- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/WriteTests.cs`, + `ProvisioningTests.cs`, `AlarmWriterTests.cs` + +**Acceptance Criteria:** +- [ ] The value writer works as follows: + - It calls `LiveWritePreparer.Prepare(values, utcNow, resolutionTicks: 10)`. + - It rejects an invalid tag name (`HistorianTagNames.IsValid`) and returns `false`. + - It reads the existing timestamps for the tag in the batch range, including tombstones, using the T0 Q11 form. + - It inserts only the rows that are not yet present, in multi-row parameterised `INSERT`s of at most the T0 Q2 + batch size, with `deleted=false` and `server_ts=utcNow()`. + - It returns `true` with no SQL when every row already existed. + - Any exception returns `false`. It never throws, and cancellation also returns `false`. +- [ ] The provisioner upserts `hist_tag` (`ts = 1970-01-01`, data type from `HistorianDataTypes.FromKind`, eng unit, + description). Both the request field types and the result tally match the SqlServer provisioner's. +- [ ] The alarm writer works as follows: + - It uses the same deterministic `event_id` as the SQL Server writer, through `AlarmEventIds.ComputeEventId` from + Sql.Common. + - It inserts events through the dedup key `(ts, event_id)`, so a redelivery is idempotent. + - `Classify` maps an unknown or non-database exception to RetryPlease, and maps permanent errors as the SqlServer + writer does. +- [ ] `WriteTests` cover: + - typed round trip for all six kinds with a non-Good status; + - a redelivered batch leaves one row and the first value; + - a tombstoned key is not resurrected by redelivery (seed a tombstone directly); + - NaN ⇒ `BadOutOfRange` Empty; + - a microsecond collision within a batch keeps the first; + - a stopped port (`Port=1`) returns `false` without throwing. +- [ ] Each regression test is falsified once. + +**Verify:** with `HISTORIAN_QUESTDB_TEST_CONNECTION` set, +`dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests --filter "FullyQualifiedName~Write|FullyQualifiedName~Provisioning|FullyQualifiedName~AlarmWriter"` +passes. + +**Steps:** +- [ ] Step 1: write `WriteTests` (red). +- [ ] Step 2: implement the value writer. +- [ ] Step 3: make `WriteTests` green, then falsify the first-wins, tombstone and NaN tests. +- [ ] Step 4: do the provisioner, then the alarm writer, each tests-first. +- [ ] Step 5: commit `feat(historian): QuestDB live writes, provisioning and alarm events` with the trailer. + +```json:metadata +{"files":["src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianValueWriter.cs","src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbTagProvisioner.cs","src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbAlarmHistorianWriter.cs","tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/WriteTests.cs","tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/ProvisioningTests.cs","tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/AlarmWriterTests.cs"],"verifyCommand":"dotnet test …QuestDb.IntegrationTests --filter Write|Provisioning|AlarmWriter (env set)","acceptanceCriteria":["insert-if-absent incl tombstones","never throws","event id deterministic + idempotent","falsified"],"modelTier":"standard","model":"sonnet","wave":3} +``` + +--- + +### Task 5: QuestDB reads and aggregates + +**Goal:** `QuestDbHistorianDataSource` serves Raw, AtTime, Events and Processed. Results are identical to SQL Server's, +except for microsecond rounding. + +**Files:** +- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianDataSource.cs`, + `QuestDbAggregateQueries.cs` +- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/ReadTests.cs` + +**Acceptance Criteria:** +- [ ] Every query filters `NOT deleted`. Every row goes through `StoredSample.FromStored`. +- [ ] **Raw:** forward and reverse, `LIMIT n+1`, and the same paging contract and continuation semantics as the + SqlServer data source (re-read from the last timestamp, inclusive). Unbounded when `maxValuesPerNode == 0`. + Unknown tag ⇒ an empty result. The tag is resolved through the `hist_tag` existence or the sample table, so the + read never creates a tag. +- [ ] **AtTime:** the result list matches the input in length and order. Neighbours come from the T0 Q9 form, and + `AtTimeEvaluator.Evaluate` decides each value. +- [ ] **Processed:** + - It uses `BucketPlan.Create` and `AggregateFamilies.FamilyOf`. + - It runs one statement per node, grouped by `(cast(ts AS LONG)*10 - @startUnixTicks) / @intervalTicks`. + - It fills the shared `BucketRow` with the same usable/good/bad/non-numeric definitions as SQL Server: usable = status + severity not Bad and kind ≠ Empty; good = usable and status < 0x40000000; non-numeric = usable and kind in (4, 5). + - `AggregateEvaluator.Evaluate` produces the values, and `BucketPlan.Assemble` fills empty buckets and handles + reversal. + - An unsupported aggregate throws `HistoryAggregateNotSupportedException`. +- [ ] **Events:** filtered by source and time range, ascending, limited to `maxEvents`. The `ContinuationPoint` is + non-null when truncated. +- [ ] Health counters (`GetHealthSnapshot`) are implemented as in the SqlServer data source (lock-guarded success/failure counters plus the last error). A failure fails only that call and is recorded in + `GetHealthSnapshot`. +- [ ] `ReadTests` cover only QuestDB-specific behaviour: + - tombstoned rows are invisible to Raw, AtTime and Processed (seed a tombstone directly); + - a whole-range Processed read from 1601 to 9999 with interval 0 does not overflow; + - a reverse paging walk with `maxValuesPerNode=1` returns every row. + The shared literals run through the conformance suite in T7. + +**Verify:** with `HISTORIAN_QUESTDB_TEST_CONNECTION` set, +`dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests --filter "FullyQualifiedName~ReadTests"` +passes. + +**Steps:** +- [ ] Step 1: write `ReadTests` (red). +- [ ] Step 2: implement Raw, then AtTime, Events and Processed. +- [ ] Step 3: temporarily compile the conformance read classes into this task's worktree. Add a throwaway concrete + subclass and a throwaway backend that seeds through direct INSERTs. Run the Raw, AtTime and Processed + conformance classes locally and get them green. +- [ ] Step 4: delete the throwaway files before committing. T7 owns the real ones. Paste the conformance run output + into the report. +- [ ] Step 5: falsify the tombstone filter and the bucket expression (an off-by-one start). +- [ ] Step 6: commit `feat(historian): QuestDB reads and aggregates (pushed down, shared evaluation)` with the trailer. + +```json:metadata +{"files":["src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianDataSource.cs","src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbAggregateQueries.cs","tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/ReadTests.cs"],"verifyCommand":"dotnet test …QuestDb.IntegrationTests --filter FullyQualifiedName~ReadTests (env set) + throwaway conformance run in report","acceptanceCriteria":["NOT deleted everywhere","bucket expr exact vs SqlServer","shared evaluators used","conformance read classes green locally"],"modelTier":"frontier","model":"opus","wave":3} +``` + +--- + +### Task 6: QuestDB HistoryUpdate (soft delete) with read-your-writes + +**Goal:** `QuestDbHistorianDataUpdater` implements all five operations through insert-only upserts and tombstones. It +returns codes identical to SQL Server's, and waits for WAL visibility before returning. + +**Files:** +- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianDataUpdater.cs`, + `QuestDbWalVisibility.cs` +- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/UpdateTests.cs`, + `TombstoneTests.cs` + +**Acceptance Criteria:** +- [ ] `UpdateDataAsync`: + 1. `HistoryUpdatePlanner.Plan(tag, mode, values, resolutionTicks: 10)` builds the plan. + 2. It reads which winner timestamps currently have a **live** row, meaning present and not deleted. + 3. It writes: + - Insert: only the absent winners; + - Replace: only the present winners; + - Update: every winner. + Each write is an `INSERT` with `deleted=false`. + 4. It builds the outcome map (present ⇒ Replaced, absent ⇒ Inserted, for rows it wrote). + 5. `HistoryUpdatePlanner.Complete(...)` produces the codes. +- [ ] `DeleteAtTimeAsync` writes a tombstone for each time with a live row, with `deleted=true`, value columns NULL and + `status=BadNoData`. Codes come from `HistoryUpdatePlanner.DeleteAtTimeResults`. +- [ ] `DeleteRawAsync` writes a tombstone for every live row in `[start, end)`. The code comes from + `HistoryUpdatePlanner.DeleteRawResult`. +- [ ] On an unknown tag, Insert and Update create the `hist_tag` row; Replace and Delete do not, and return the shared + unknown-tag codes. +- [ ] `QuestDbWalVisibility.WaitAsync(table, ct)` works as follows: + - It polls the T0 Q4 query until the table's applied transaction is at least the sequencer transaction captured + after our write. + - It gives up after `VisibilityTimeout` and logs a Warning, without changing the codes. `VisibilityTimeout = 0` + skips the wait. + - Every updater operation calls it once, after writing. +- [ ] `UpdateTests` cover the five operations' codes (QuestDB-specific checks only; the literal codes run through + conformance in T7) and that a read made right after `UpdateDataAsync` returns sees the change. +- [ ] `TombstoneTests` cover: + - delete then Raw read ⇒ the row is gone; + - delete, then a `QuestDbHistorianValueWriter` redelivery of the same sample ⇒ the row stays gone (this needs T4's + writer: write the test against the interface, and if T4 is not merged yet, seed a tombstone and assert the + writer's existence query counts it); + - insert over a tombstone ⇒ `GoodEntryInserted` and a visible row. + +**Verify:** with `HISTORIAN_QUESTDB_TEST_CONNECTION` set, +`dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests --filter "FullyQualifiedName~UpdateTests|FullyQualifiedName~TombstoneTests"` +passes. + +**Steps:** +- [ ] Step 1: write the tests (red). +- [ ] Step 2: implement `QuestDbWalVisibility`. +- [ ] Step 3: implement the updater. +- [ ] Step 4: get to green. Falsify the "live row" filter by dropping `NOT deleted` from the existence check, and the + visibility wait by setting the timeout to 0 in the read-after-write test. +- [ ] Step 5: commit `feat(historian): QuestDB HistoryUpdate with soft delete and read-your-writes` with the trailer. + +```json:metadata +{"files":["src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianDataUpdater.cs","src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbWalVisibility.cs","tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/UpdateTests.cs","tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/TombstoneTests.cs"],"verifyCommand":"dotnet test …QuestDb.IntegrationTests --filter UpdateTests|TombstoneTests (env set)","acceptanceCriteria":["insert-only upserts, no UPDATE/DELETE","codes from shared planner","tombstones never resurrected","read-your-writes wait"],"modelTier":"standard","model":"sonnet","wave":3} +``` + +--- + +### Task 7: Provider, Host wiring and the conformance suite on QuestDB + +**Goal:** `Backend=QuestDb` resolves, validates and builds all five seams. The full conformance suite passes against +QuestDB. + +**Files:** +- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianProvider.cs` +- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/QuestDbConformanceBackend.cs`, + `QuestDbConformance.cs` (one concrete subclass per conformance class) +- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests/QuestDbHistorianProviderTests.cs` +- Modify: + - `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/HistorianBackendRegistry.cs` (add a `QuestDbBackendName` + const, `BindQuestDb(IConfiguration)` and a `Factories` entry); + - `ServerHistorianOptionsValidator.cs` (a QuestDb block before the `if (!options.IsGatewayBackend) return;` line, + mirroring the SqlServer block); + - `src/Server/ZB.MOM.WW.OtOpcUa.Host/ZB.MOM.WW.OtOpcUa.Host.csproj` (a ProjectReference); + - `tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests/Configuration/HistorianBackendRegistryTests.cs`; + - `tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/ServerHistorianOptionsValidatorTests.cs`; + - `docker-dev/docker-compose.yml`: in the central-1 and central-2 `ServerHistorian__*` blocks, add + `ServerHistorian__QuestDb__ConnectionString: "${OTOPCUA_HISTORIAN_QUESTDB:-}"` + plus a comment line naming the overlay. + +**Acceptance Criteria:** +- [ ] The provider has `Name="QuestDb"`, `SupportedKinds` = all six kinds as a FrozenSet, and one lazily built shared + connection and bootstrapper across the seams. The constructor does no I/O and no validation, like the SqlServer + provider. `CreateUpdater` returns the soft-delete updater. +- [ ] The registry resolves `QuestDb` case-insensitively. +- [ ] The validator, when the backend is consumed and is `QuestDb`: + - reports every `QuestDbHistorianOptions.Validate()` error, naming the key; + - does not require the Gateway keys; + - ignores an invalid `TablePrefix` when not consumed. +- [ ] Registry and validator tests mirror the five SqlServer ones (lines 71 and 319–368). +- [ ] The QuestDB integration project runs the **entire** conformance suite through `QuestDbConformanceBackend`: + - `ResolutionTicks = 10`; + - seeding is direct INSERTs into the prefixed tables; + - `ReadBackAsync` filters `NOT deleted`. + Every class passes. Any named override has a doc comment and a ledger ruling. + +**Verify:** +- With both environment variables set, the SqlServer and QuestDB integration projects both pass. The report shows the + conformance class counts side by side. +- `dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests --filter "FullyQualifiedName~Historian"` passes. +- `dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests --filter "FullyQualifiedName~Historian"` passes. + +**Steps:** +- [ ] Step 1: write the provider unit tests (red), then the provider. +- [ ] Step 2: write the registry and validator tests (red), then the wiring. +- [ ] Step 3: write `QuestDbConformanceBackend` and the subclasses, and run the full suite. Fix any divergence in the + QuestDB code, even though those files belong to T4–T6; this task is the integration owner in W4. +- [ ] Step 4: make the compose env edit. +- [ ] Step 5: build the solution, then commit `feat(historian): QuestDb provider + Host wiring; conformance suite green + on both backends` with the trailer. + +```json:metadata +{"files":["src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianProvider.cs","tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/QuestDbConformanceBackend.cs","tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/QuestDbConformance.cs","src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/HistorianBackendRegistry.cs","src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/ServerHistorianOptionsValidator.cs","src/Server/ZB.MOM.WW.OtOpcUa.Host/ZB.MOM.WW.OtOpcUa.Host.csproj","docker-dev/docker-compose.yml"],"verifyCommand":"both IT projects green with env set; Host historian filters green","acceptanceCriteria":["provider shape","registry + consumer-gated validator","full conformance suite green on QuestDB","compose env"],"modelTier":"standard","model":"sonnet","wave":4} +``` + +--- + +### Task 8: Documentation + +**Goal:** the docs describe the QuestDB backend as built, and every claim is checked against the merged code. + +**Files:** `docs/Historian.md` (a new §"QuestDB backend"), `docs/Configuration.md` (the `ServerHistorian:QuestDb` +table), `CLAUDE.md` (one short bullet in the Pluggable backends subsection), `docker-dev/README.md` (the overlay +recipe), and the spec (an as-built pass) + +**Acceptance Criteria:** +- [ ] Every `ServerHistorian:QuestDb` key is documented with its default and validation rule. +- [ ] The QuestDB-specific behaviour is documented: + - soft delete and tombstones; + - insert-if-absent including tombstones; + - microsecond truncation and collisions; + - WAL visibility and the read-your-writes wait; + - TTL; + - the overlay recipe and `HISTORIAN_QUESTDB_TEST_CONNECTION`. +- [ ] Each statement is cross-checked against the code, and any spec-versus-code disagreement is fixed in the spec + inline and listed in the report. + +**Verify:** the controller reads the diff, and `grep -n "QuestDb" docs/Configuration.md` shows the table. + +```json:metadata +{"files":["docs/Historian.md","docs/Configuration.md","CLAUDE.md","docker-dev/README.md","docs/plans/2026-10-10-questdb-historian-design.md"],"verifyCommand":"grep -n QuestDb docs/Configuration.md","acceptanceCriteria":["keys documented","QuestDB-specific semantics documented","claims cross-checked"],"modelTier":"standard","model":"sonnet","wave":5} +``` + +--- + +### Final review, then Task 9: live gate on docker-dev (controller) + +**Final review:** a **fable** reviewer reads `git diff master...feat/questdb-historian` together with the spec. It must +report PASS or FAIL on each of these checks: +1. SqlServer behaviour is unchanged: identical conformance results, and no deleted SqlServer test. +2. No QuestDB `UPDATE` or `DELETE` is issued on data. +3. `NOT deleted` is present in every read path. +4. Every new status literal is a named const. +5. No package or version changes. +6. The bucket expression is exact. +7. #491 still holds: no Self/Context access after an await. + +Then one fix dispatch (opus) and a scoped re-review (sonnet), followed by adjudication. + +**Task 9: live gate (controller only).** + +**Goal:** prove `Backend=QuestDb` end to end on the docker-dev rig, against the overlay `questdb` service, with captured +output for every leg. + +> **USER-ORDERED GATE — NON-SKIPPABLE.** This task was requested by the user in the current conversation. It MUST NOT +> be closed by walking around it, by declaring it "verified inline", or by substituting a cheaper check. Close only +> after every item in `acceptanceCriteria` has been re-validated independently, with output captured. + +**Files:** `docs/plans/2026-10-10-questdb-historian-live-gate.md` + +**Acceptance Criteria:** run the six legs of spec §7 on docker-dev with `OTOPCUA_HISTORIAN_BACKEND=QuestDb`. Each leg +needs captured output. + +**Verify:** the evidence doc has each leg's CLI output plus the direct QuestDB query output. + +**Steps:** +1. Bring up the overlay `questdb`. +2. Rebuild the central image and recreate the central nodes with `OTOPCUA_HISTORIAN_ENABLED=true`, + `OTOPCUA_HISTORIAN_BACKEND=QuestDb` and `OTOPCUA_CONTHIST_ENABLED=true`. +3. Historize FastUInt1, a string calculation tag and `HR5`. Add a UNS reference. Mint the two gate API keys, as in the + SQL Server gate recipe (`docs/plans/2026-10-09-pluggable-historian-live-gate.md`), and deploy. +4. Run legs 1–6 with direct `psql` or REST queries. QuestDB REST is `curl -G localhost:19000/exec --data-urlencode "query=…"`. +5. Write the evidence doc. +6. Restore the rig: recreate the centrals with defaults, revert the config DB, redeploy, `down` the overlay, and stop + the fixtures. + +```json:metadata +{"files":["docs/plans/2026-10-10-questdb-historian-live-gate.md"],"verifyCommand":"Client.CLI historyread/historyupdate against opc.tcp://localhost:4840 + QuestDB REST /exec on :19000","acceptanceCriteria":["Primary-only typed recording one row per value","raw+UNS HistoryRead/Processed identical and match QuestDB","HistoryUpdate codes + delete not resurrected by redelivery + WriteOperate-only and anonymous denied","failover 1x","Modbus loss: BadCommunicationError then BadNoCommunication marker then Good","TTL visible in tables() metadata"],"userGate":true,"tags":["user-gate"],"modelTier":"frontier","model":"controller","wave":5} +``` + +--- + +## Finish + +- Merge to `master` and push only with the user's go-ahead. +- After merging, append the scadaproj `sister-project-history/otopcua.md` entry on scadaproj `main` and push it. +- Close #666. diff --git a/docs/plans/2026-10-10-questdb-historian-plan.md.tasks.json b/docs/plans/2026-10-10-questdb-historian-plan.md.tasks.json new file mode 100644 index 00000000..5d35edc6 --- /dev/null +++ b/docs/plans/2026-10-10-questdb-historian-plan.md.tasks.json @@ -0,0 +1,96 @@ +{ + "planPath": "docs/plans/2026-10-10-questdb-historian-plan.md", + "tasks": [ + { + "id": 0, + "subject": "Task 0: QuestDB spike, overlay compose and spec amendments", + "status": "pending", + "description": "**Goal:** prove every QuestDB behaviour the design assumes, against a pinned image, before any code is built on it.\nThe findings are written into spec \u00a72\u2013\u00a74.\n\n**Files:**\n- Create: `docker-dev/docker-compose.questdb.yml`\n- Create: `docs/plans/2026-10-10-questdb-spike.md`\n- Modify: `docs/plans/2026-10-10-questdb-historian-design.md` (\u00a72\u2013\u00a74: replace assumptions with confirmed forms, and mark\n each one \"T0-confirmed\")\n\n**Acceptance Criteria:**\n- [ ] The overlay pins an exact `questdb/questdb:` tag. The tag's release date is at least two weeks before\n 2026-10-10, and is recorded in the spike doc.\n- [ ] The overlay exposes `18812:8812` and `19000:9000`, uses a named volume `otopcua-questdb-data`, and sets the\n container name `otopcua-dev-questdb-1`. It is brought up with\n `docker compose -f docker-dev/docker-compose.yml -f docker-dev/docker-compose.questdb.yml up -d questdb`.\n- [ ] The spike doc records a working Npgsql connection string and answers Q1\u2013Q12 below, each with the exact SQL or\n C# that worked and its observed output.\n- [ ] Spec \u00a72.1 has the confirmed DDL, \u00a72.3 the confirmed visibility-wait query, \u00a73 the confirmed bucket-statistics and\n neighbour query forms, and \u00a74 the confirmed connection-string keywords.\n\n**Verify:** `docker exec otopcua-dev-questdb-1 true` exits 0. The spike harness output is pasted into the spike doc.\n\n**Steps:**\n\n- [ ] **Step 1: Create the overlay.**\n\n```yaml\n# docker-dev/docker-compose.questdb.yml \u2014 optional QuestDB fixture for the QuestDB historian backend.\n# Up: docker compose -f docker-dev/docker-compose.yml -f docker-dev/docker-compose.questdb.yml up -d questdb\nservices:\n questdb:\n image: questdb/questdb: # T0 pins an exact tag \u2265 2 weeks old\n container_name: otopcua-dev-questdb-1\n ports:\n - \"18812:8812\" # PG wire\n - \"19000:9000\" # web console / REST\n volumes:\n - otopcua-questdb-data:/var/lib/questdb\n healthcheck:\n test: [\"CMD-SHELL\", \"curl -fs http://localhost:9003/status || exit 1\"]\n interval: 10s\n timeout: 5s\n retries: 12\nvolumes:\n otopcua-questdb-data:\n```\n\n- [ ] **Step 2: Write a throwaway C# harness** in the scratchpad, not in the repo. It is a console project that\n references `Npgsql`, restored from the existing central pin via a temporary `Directory.Packages.props` copy, or\n run with `dotnet run` inside a scratch folder under the repo with `ManagePackageVersionsCentrally`. Answer each\n question with real output:\n - **Q1.** The connection-string keywords that work: test `ServerCompatibilityMode=NoTypeLoading` with and without\n it. Also the parameter types needed for `TIMESTAMP` (UTC `DateTime`), `BYTE`, `SYMBOL`, `VARCHAR`, `BOOLEAN` and\n `LONG`.\n - **Q2.** Whether a parameterised multi-row `INSERT INTO t VALUES (\u2026),(\u2026)` works, and the largest row count per\n statement that works reliably. Pick the batch size: \u2264 1000 rows, capped by any parameter limit found.\n - **Q3.** `CREATE TABLE \u2026 WAL DEDUP UPSERT KEYS(ts, tag)`: re-inserting the same key replaces the row. Show\n before and after.\n - **Q4.** The `wal_tables()` columns. After an insert, how to know the transaction we produced and wait until it is\n applied (`writerTxn >= sequencerTxn`). Measure the typical lag over 20 runs.\n - **Q5.** TTL: the syntax at create (`TTL n DAYS`), and `ALTER TABLE \u2026 SET TTL n DAYS`. Where TTL shows up in\n `tables()` metadata.\n - **Q6.** Timestamp precision through Npgsql: a value with 100 ns ticks (`\u20261234567`) comes back as `\u2026123456` (truncated)\n or `\u2026123457` (rounded)? Confirm that our pre-truncation makes the round trip exact.\n - **Q7.** Two connections run `CREATE TABLE IF NOT EXISTS` for the same table at once, 20 times. Record any error\n text, and the catch rule that makes it idempotent.\n - **Q8.** Bucket statistics: one `GROUP BY` over the expression `(cast(ts AS LONG)*10 - $1) / $2`, with\n `sum(CASE WHEN \u2026 THEN 1 ELSE 0 END)`, `avg`, `min`, `max`, `sum`. Plus the first and last usable sample per bucket,\n with its value and time. Try `first()`/`last()` with a filter, or a window `row_number() OVER (PARTITION BY b\n ORDER BY ts)`. Plus arg-min/arg-max time, via a window function or a second statement.\n - **Q9.** AtTime neighbour lookup: chunked `LIMIT 1` seeks (`ts <= t ORDER BY ts DESC LIMIT 1` and\n `ts > t ORDER BY ts LIMIT 1`, combined with `UNION ALL` per time) versus `ASOF JOIN`. Time both for 100 timestamps\n and pick one.\n - **Q10.** VARCHAR and SYMBOL accept a 400-character string, and `SYMBOL` capacity is fine for tens of thousands of\n tags.\n - **Q11.** An existing-timestamp lookup for a batch: `WHERE tag=$1 AND ts >= $2 AND ts <= $3` (a range), or an `IN`\n list. Pick the one that works with parameters.\n - **Q12.** `DROP TABLE IF EXISTS` works for test cleanup, and `tables()` lists the tables with a given prefix.\n\n- [ ] **Step 3: Write the spike doc** with the image tag and date, the connection string, and Q1\u2013Q12 (SQL, output and\n decision for each).\n- [ ] **Step 4: Amend the spec \u00a72\u2013\u00a74** in place with the confirmed forms. If any assumption failed, write the\n replacement design inline and add a ledger ruling. If the failure breaks a user decision, stop and ask.\n- [ ] **Step 5: Commit**\n\n```bash\ngit add docker-dev/docker-compose.questdb.yml docs/plans/2026-10-10-questdb-spike.md docs/plans/2026-10-10-questdb-historian-design.md\ngit commit -m \"spike(historian): QuestDB behaviours confirmed against pinned image; spec \u00a72\u2013\u00a74 amended\n\nClaude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe\"\n```\n\n```json:metadata\n{\"files\":[\"docker-dev/docker-compose.questdb.yml\",\"docs/plans/2026-10-10-questdb-spike.md\",\"docs/plans/2026-10-10-questdb-historian-design.md\"],\"verifyCommand\":\"docker exec otopcua-dev-questdb-1 true\",\"acceptanceCriteria\":[\"pinned image \u22652 weeks old\",\"overlay ports 18812/19000\",\"Q1\u2013Q12 answered with output\",\"spec \u00a72\u2013\u00a74 amended\"],\"modelTier\":\"frontier\",\"model\":\"controller\",\"wave\":0}\n```\n\n---" + }, + { + "id": 1, + "subject": "Task 1: Extract `Driver.Historian.Sql.Common` (no behaviour cha", + "status": "pending", + "description": "**Goal:** move every database-independent piece of `Driver.Historian.SqlServer` into a new shared library. The SQL\nServer backend then calls it, with byte-identical behaviour.\n\n**Files:**\n- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.csproj`\n- Create in that project: `HistorianStatusCodes.cs`, `StatusBits.cs`, `HistorianInfoBits.cs`, `HistorianDataTypes.cs`,\n `SampleColumns.cs`, `HistorianSampleRow.cs`, `StoredSample.cs`, `BucketRow.cs`, `AggregateFamily.cs`,\n `AggregateEvaluator.cs`, `AtTimeEvaluator.cs`, `BucketPlan.cs`, `LiveWritePreparer.cs`, `HistoryUpdatePlanner.cs`,\n `HistorianTagNames.cs`, `BootstrapFailureCache.cs`, `HistorianSchemaMissingException.cs`, `AlarmEventIds.cs`\n- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/` (csproj + tests)\n- Modify: every file in `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/` that contained the moved code\n (`SqlServerHistorianDataSource.cs`, `SqlAggregateQueries.cs`, `SqlHistorianDataTypes.cs` (delete),\n `SqlHistorianStatusCodes.cs` (keep only SQL Server-only codes, or delete if none), `SampleRowMapper.cs`,\n `SqlServerHistorianValueWriter.cs`, `SqlServerHistorianDataUpdater.cs`, `SqlServerTagCatalog.cs`,\n `SqlServerSchemaBootstrapper.cs`, the csproj)\n- Modify: the SqlServer unit tests that tested moved code. Move them into `Sql.Common.Tests` under the same test name.\n- Modify: `ZB.MOM.WW.OtOpcUa.slnx` (add both new projects alphabetically, next to lines 30 and 104)\n- Modify: `tests/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions.Tests/ZB.MOM.WW.OtOpcUa.Core.Abstractions.Tests.csproj` (add a\n ProjectReference to Sql.Common)\n\n**Acceptance Criteria:**\n- [ ] `Sql.Common` has no reference to `Microsoft.Data.SqlClient` or any other database client. Its references are\n Core.Abstractions, Server/Runtime (only if needed for `HistorizationValue`) and\n `Microsoft.Extensions.Logging.Abstractions`.\n- [ ] The public API matches the signatures in Step 1 exactly. T2\u2013T7 depend on them.\n- [ ] The SqlServer integration suite passes with the same test count as before the change, which is recorded in the\n report. The SqlServer unit suite plus the moved tests in `Sql.Common.Tests` has the same total count.\n- [ ] New `Sql.Common.Tests` unit tests cover `AtTimeEvaluator`, `AggregateEvaluator`, `BucketPlan`, `LiveWritePreparer`\n and `HistoryUpdatePlanner` directly, using the literals from the SqlServer integration tests. Each one is\n falsified once.\n- [ ] StatusCodeParity passes, and its constant count does not drop.\n\n**Verify:**\n- `dotnet build ZB.MOM.WW.OtOpcUa.slnx` gives 0 errors.\n- `dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests` passes.\n- `dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.Tests` passes.\n- With `HISTORIAN_SQL_TEST_CONNECTION` set, `dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests`\n passes with the same count as before.\n- `dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions.Tests --filter \"FullyQualifiedName~StatusCodeParity\"`\n passes.\n\n**Steps:**\n\n- [ ] **Step 0: Record the baseline.** Run both SqlServer test projects and the parity filter, and write the exact\n pass counts into the report.\n\n- [ ] **Step 1: Create the shared API.** Namespace: `ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common`. Each moved body is\n transplanted verbatim from the source named in the comment; only names and visibility change.\n\n```csharp\n// HistorianStatusCodes.cs \u2014 from SqlHistorianStatusCodes + SqlHistorianUpdateStatusCodes (all consts, same values)\npublic static class HistorianStatusCodes\n{\n public const uint Good = 0x00000000u, GoodEntryInserted = 0x00A20000u, GoodEntryReplaced = 0x00A30000u;\n public const uint BadNoData = 0x809B0000u, BadEntryExists = 0x809F0000u, BadNoEntryExists = 0x80A00000u;\n public const uint BadInvalidTimestamp = 0x80230000u, BadAggregateInvalidInputs = 0x80D60000u;\n public const uint UncertainDataSubNormal = 0x40A40000u, BadOutOfRange = 0x803C0000u, BadInvalidArgument = 0x80AB0000u;\n}\n// StatusBits.cs / HistorianInfoBits.cs \u2014 moved verbatim from SqlHistorianStatusCodes.cs (made public)\n// HistorianDataTypes.cs \u2014 from SqlHistorianDataTypes: DefaultForEmpty, byte FromKind(HistorianValueKind)\n// SampleColumns.cs \u2014 record struct SampleColumns(byte ValueKind, double? NumValue, long? IntValue, string? StrValue)\n// + static class SampleColumnMapper { ToColumns(HistorianValue); FromColumns(byte,double?,long?,string?);\n// int ToDbStatus(uint); uint FromDbStatus(int); long ToLongStatus(uint) => status; uint FromLongStatus(long) }\n// HistorianSampleRow.cs \u2014 record struct HistorianSampleRow(DateTime TimeUtc, HistorianValue Value, uint StatusCode) (was SqlSampleRow)\npublic readonly record struct StoredSample(DateTime TimeUtc, HistorianValue Value, uint Status)\n{\n public bool IsUsable { get; } // verbatim rule from SqlServerHistorianDataSource\n public bool IsGood { get; }\n /// Applies the quality clamp (NULL value + non-Bad \u21d2 BadNoData) \u2014 from ReadStored line 533.\n public static StoredSample FromStored(DateTime timeUtc, HistorianValue value, uint status);\n}\npublic sealed class BucketRow // fields verbatim from the private BucketRow, as public get/set properties\n{\n public long Bucket { get; set; } public int Total { get; set; } public int Usable { get; set; }\n public int Good { get; set; } public int Bad { get; set; } public int NonNumeric { get; set; }\n public double? Avg { get; set; } public double? Min { get; set; } public double? Max { get; set; } public double? Sum { get; set; }\n public HistorianValue First { get; set; } public HistorianValue Last { get; set; }\n public double? Extreme { get; set; } public DateTime? ExtremeTime { get; set; }\n}\npublic enum AggregateFamily { Simple, StartEnd, ActualTime }\npublic static class AggregateFamilies { public static AggregateFamily? FamilyOf(HistoryAggregateType a); } // from SqlAggregateQueries.FamilyOf\npublic static class AggregateEvaluator\n{\n public static DataValueSnapshot Evaluate(HistoryAggregateType a, BucketRow r, DateTime bucketStart, DateTime now); // verbatim\n public static DataValueSnapshot NoData(DateTime t, DateTime now); // verbatim\n}\npublic static class AtTimeEvaluator\n{\n public static DataValueSnapshot Evaluate(DateTime t, StoredSample? prev, StoredSample? next, DateTime serverTime, DateTime now); // verbatim AtTimeValue\n}\npublic sealed class BucketPlan // from ReadProcessedAsync lines 355-370 and 395-408\n{\n public const int MaxBuckets = 100_000;\n public DateTime StartUtc { get; } public DateTime EndUtc { get; } public bool Reversed { get; }\n public long IntervalTicks { get; } public long BucketCount { get; }\n public long StartUnixTicks => (StartUtc - DateTime.UnixEpoch).Ticks; // for QuestDB's (cast(ts AS LONG)*10 - start)/interval\n /// BucketCount > MaxBuckets (param name \"interval\") \u2014 same as today.\n public static BucketPlan Create(DateTime startUtc, DateTime endUtc, TimeSpan interval);\n public DateTime BucketStart(long index);\n /// One snapshot per bucket in window order; missing buckets get ; reversed if the window was.\n public IReadOnlyList Assemble(IReadOnlyDictionary byBucket, Func empty);\n}\npublic static class LiveWritePreparer // from SqlServerHistorianValueWriter.Prepare, plus a resolution parameter\n{\n /// 1 = keep 100 ns (SQL Server); 10 = truncate to \u00b5s BEFORE de-duplication (QuestDB).\n public static List Prepare(IReadOnlyList values, Func utcNow, long resolutionTicks = 1);\n}\npublic enum UpsertOutcome { Inserted, Replaced }\npublic sealed class UpdatePlan\n{\n public uint?[] Results { get; } // pre-decided codes (invalid ts, NaN, superseded duplicate\u2026); null = decided by the backend\n public IReadOnlyList<(int Index, HistorianSampleRow Row)> Winners { get; } // rows to send, de-duplicated per mode\n}\npublic static class HistoryUpdatePlanner // pre/post passes from SqlServerHistorianDataUpdater.UpdateDataAsync lines 56-108\n{\n public static UpdatePlan Plan(string tag, HistoryUpdateMode mode, IReadOnlyList values, long resolutionTicks = 1);\n /// ticks of each winner that the backend actually wrote \u2192 Inserted/Replaced; absent \u21d2 not written.\n public static IReadOnlyList Complete(UpdatePlan plan, HistoryUpdateMode mode, IReadOnlyDictionary outcomeByTick);\n public static IReadOnlyList DeleteAtTimeResults(IReadOnlyList times, ISet deletedTicks, bool tagKnown);\n public static uint DeleteRawResult(bool tagKnown, long rowsAffected);\n}\npublic static class HistorianTagNames { public const int MaxLength = 400; public static bool IsValid(string? name); } // from SqlServerTagCatalog.IsValidName\npublic sealed class BootstrapFailureCache // the 5 s negative-cache mechanism from SqlServerSchemaBootstrapper\n{\n public BootstrapFailureCache(TimeProvider clock, TimeSpan duration);\n public void ThrowIfCached(); public void Record(Exception ex); public void Clear();\n}\n// HistorianSchemaMissingException.cs \u2014 moved from SqlServerHistorianOptions.cs (public sealed, ctor(string message))\n// AlarmEventIds.cs \u2014 public static string ComputeEventId(...) moved verbatim from SqlServerAlarmHistorianWriter (same signature)\n```\n\n- [ ] **Step 2: Point the SqlServer backend at the shared lib.**\n - Replace each moved body with a call into `Sql.Common`.\n - Delete the now-empty SqlServer classes.\n - Keep `SqlAggregateQueries`' SQL text and `ReadBucket`, now filling the shared `BucketRow`.\n - Keep `SampleRowMapper`'s TVP builders, which take `HistorianSampleRow`.\n- [ ] **Step 3: Move the SqlServer unit tests for the moved code** into `Sql.Common.Tests`, keeping test names and\n literals: `StatusCodeOracleTests`, `ValueWriterPrepareTests`, the column round-trip parts of\n `SampleRowMapperTests`, and the data-type table test. The TVP parts stay in SqlServer.Tests.\n- [ ] **Step 4: Add direct unit tests that did not exist before.** For example:\n\n```csharp\n[Fact] public void AtTime_inside_an_outage_returns_the_bad_neighbours_code_with_no_value()\n{\n var t0 = new DateTime(2026,10,9,8,0,0,DateTimeKind.Utc);\n var prev = new StoredSample(t0, HistorianValue.Empty, 0x80310000u);\n var next = new StoredSample(t0.AddSeconds(10), HistorianValue.FromDouble(5), 0);\n var r = AtTimeEvaluator.Evaluate(t0.AddSeconds(5), prev, next, t0, t0.AddHours(1));\n r.StatusCode.ShouldBe(0x80310000u); r.Value.ShouldBeNull();\n}\n[Fact] public void BucketPlan_rejects_more_than_100k_buckets()\n => Should.Throw(() =>\n BucketPlan.Create(DateTime.UnixEpoch, DateTime.UnixEpoch.AddSeconds(100_001), TimeSpan.FromSeconds(1)));\n[Fact] public void Prepare_truncates_to_microseconds_before_dedup_when_resolution_is_10()\n{\n var t = new DateTime(2026,10,9,8,0,0,DateTimeKind.Utc).AddTicks(1234567);\n var rows = LiveWritePreparer.Prepare([new(t, HistorianValue.FromDouble(1), 0), new(t.AddTicks(3), HistorianValue.FromDouble(2), 0)], () => t, 10);\n rows.Count.ShouldBe(1); rows[0].TimeUtc.Ticks.ShouldBe(t.Ticks - t.Ticks % 10); rows[0].Value.Number.ShouldBe(1);\n}\n```\n\n Add the matching `AggregateEvaluator` cases, using the 8-row ProcessedRead literals from\n `ProcessedReadTests.Numeric_aggregates_over_the_fixture`. Build the `BucketRow`s by hand from tag A's samples.\n- [ ] **Step 5: Wire the solution and parity test.** Add both projects to the slnx, and the ProjectReference to the\n Core.Abstractions.Tests csproj.\n- [ ] **Step 6: Run the Verify commands.** Counts must equal the Step 0 baseline, plus the new direct tests. Falsify\n each new direct test once by reverting the rule it pins.\n- [ ] **Step 7: Commit** with `refactor(historian): extract Driver.Historian.Sql.Common from the SQL Server backend (no\n behaviour change)` and the Claude-Session trailer.\n\n```json:metadata\n{\"files\":[\"src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/\",\"tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/\",\"src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/\",\"tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.Tests/\",\"ZB.MOM.WW.OtOpcUa.slnx\",\"tests/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions.Tests/ZB.MOM.WW.OtOpcUa.Core.Abstractions.Tests.csproj\"],\"verifyCommand\":\"dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests (env set) \u2014 same count as baseline\",\"acceptanceCriteria\":[\"no DB client ref in Sql.Common\",\"public API exactly as Step 1\",\"SqlServer suites same counts\",\"direct evaluator tests falsified\",\"parity green\"],\"modelTier\":\"frontier\",\"model\":\"opus\",\"wave\":1}\n```\n\n---", + "blockedBy": [ + 0 + ] + }, + { + "id": 2, + "subject": "Task 2: Conformance suite (shared source), run against SQL Serv", + "status": "pending", + "description": "**Goal:** one backend-neutral suite of hand-computed behaviour tests, compiled into the SqlServer integration project\nand passing there. T7 later compiles the same suite into the QuestDB project.\n\n**Files:**\n- Create: `tests/Drivers/Historian.Conformance/IConformanceBackend.cs`, `ConformanceData.cs`,\n `RawReadConformance.cs`, `AtTimeConformance.cs`, `ProcessedConformance.cs`, `QualityContractConformance.cs`,\n `LiveWriteConformance.cs`, `UpdateConformance.cs`, `TagNameConformance.cs`\n- Modify: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/*.csproj` (add\n `` and a ProjectReference to Sql.Common)\n- Create in the SqlServer IT project: `SqlServerConformanceBackend.cs`, `SqlServerConformance.cs` (one concrete\n subclass per conformance class)\n- Modify in the SqlServer IT project: remove from the original files the generic tests that moved into the suite\n (`RawReadTests`, `AtTimeReadTests`, `ProcessedReadTests`, `QualityContractTests`, and the generic parts of\n `WriteTests`, `UpdateTests` and `LongNameTests`). SQL Server-specific tests stay where they are.\n\n**Acceptance Criteria:**\n- [ ] Every moved test keeps its method name and literal expectations. `git log -p` shows a move, not a rewrite.\n- [ ] The SqlServer IT total count after the change is at least the pre-change count, and the report has the before\n and after count.\n- [ ] Suite code references only `IConformanceBackend` and Core.Abstractions types. There is no `SqlServer*` type and\n no T-SQL in `tests/Drivers/Historian.Conformance`.\n- [ ] Timestamp assertions go through `B.Round(t)`.\n- [ ] Any test that cannot hold for every backend is a `virtual` member with a doc comment naming the difference.\n None should be needed for SQL Server.\n\n**Verify:** with `HISTORIAN_SQL_TEST_CONNECTION` set, `dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests`\npasses with a count of at least the baseline. Without the variable, every test skips.\n\n**Steps:**\n\n- [ ] **Step 1: Define the backend contract.**\n\n```csharp\nnamespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Conformance;\npublic interface IConformanceBackend : IAsyncDisposable\n{\n /// 1 for SQL Server (100 ns), 10 for QuestDB (\u00b5s).\n long ResolutionTicks { get; }\n DateTime Round(DateTime t) => new(t.Ticks - t.Ticks % ResolutionTicks, DateTimeKind.Utc);\n IHistorianDataSource CreateDataSource(TimeProvider? clock = null);\n IHistorianValueWriter CreateValueWriter(Func? utcNow = null);\n IHistorianDataUpdater CreateUpdater();\n /// Writes rows directly in the backend's schema (never through the writer), status verbatim.\n Task SeedAsync(string tag, params (DateTime T, HistorianValue V, uint Status)[] samples);\n /// Live (non-deleted) rows for a tag, ascending, read directly from storage.\n Task> ReadBackAsync(string tag);\n}\npublic abstract class ConformanceBase : IAsyncLifetime\n{\n protected IConformanceBackend B { get; private set; } = null!;\n protected abstract Task CreateBackendAsync(); // calls fixture.SkipIfUnavailable()\n public async ValueTask InitializeAsync() => B = await CreateBackendAsync();\n public async ValueTask DisposeAsync() => await B.DisposeAsync();\n}\n```\n\n `ConformanceData` holds `T0`, `At(seconds)`, the status consts and `SeedAAsync(B)`, verbatim from `ReadTestDb`.\n- [ ] **Step 2: Move the generic tests** listed in the T1 inventory (Raw, AtTime, Processed literals, the quality\n contract, live-write redelivery and first-wins, update result codes, tag names over 400 characters) into the\n suite classes. Write each one as an abstract class `RawReadConformance : ConformanceBase` with `[Fact]` methods.\n- [ ] **Step 3: Implement `SqlServerConformanceBackend`.** It reuses `SqlHistorianFixture`, a unique schema and the\n existing seeding SQL from `ReadTestDb`. Add one concrete class per suite:\n `[Collection(SqlHistorianCollection.Name)] public sealed class SqlServerRawReadConformance(SqlHistorianFixture fx) : RawReadConformance`.\n- [ ] **Step 4: Run the tests and compare counts.** Commit with\n `test(historian): backend conformance suite (shared source), run against SQL Server` and the trailer.\n\n```json:metadata\n{\"files\":[\"tests/Drivers/Historian.Conformance/\",\"tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/\"],\"verifyCommand\":\"dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests (env set) \u2265 baseline count\",\"acceptanceCriteria\":[\"tests moved by name with same literals\",\"count \u2265 baseline\",\"suite is backend-neutral\",\"Round() for timestamps\"],\"modelTier\":\"standard\",\"model\":\"sonnet\",\"wave\":2}\n```\n\n---", + "blockedBy": [ + 1 + ] + }, + { + "id": 3, + "subject": "Task 3: QuestDB skeleton (options, connection, bootstrapper, fi", + "status": "pending", + "description": "**Goal:** a building `Driver.Historian.QuestDb` project. It has validated options, a pooled Npgsql connection, an\nidempotent and concurrency-safe schema bootstrapper with TTL, and an env-gated integration fixture.\n\n**Files:**\n- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.csproj`\n (references Core.Abstractions, Core.AlarmHistorian, Server/Runtime and Sql.Common; packages `Npgsql` and\n `Microsoft.Extensions.Logging.Abstractions`; InternalsVisibleTo the two QuestDb test assemblies)\n- Create: `QuestDbHistorianOptions.cs`, `QuestDbConnection.cs`, `QuestDbTables.cs`, `QuestDbSchemaBootstrapper.cs`\n- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests/` (csproj, `QuestDbHistorianOptionsTests.cs`,\n `QuestDbTablesTests.cs`)\n- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/` (csproj with the\n conformance-include line and a ProjectReference to Sql.Common, `QuestDbFixture.cs`, `BootstrapTests.cs`)\n- Modify: `ZB.MOM.WW.OtOpcUa.slnx`, and the Core.Abstractions.Tests csproj (ProjectReference to the QuestDb project)\n\n**Acceptance Criteria:**\n- [ ] `QuestDbHistorianOptions` has `ConnectionString` (default `\"\"`), `TablePrefix` (`\"hist_\"`, regex\n `^[A-Za-z0-9_]+$`), `RetentionDays` (`int?`, \u2265 1), `CommandTimeout` (30 s, > 0), `AutoCreateSchema` (`true`) and\n `VisibilityTimeout` (5 s, \u2265 0). `Validate()` returns error strings that name the key\n `ServerHistorian:QuestDb:`.\n- [ ] `QuestDbTables` produces the three `CREATE TABLE IF NOT EXISTS` statements, as confirmed by T0, for a given prefix\n and retention. Unit tests pin the exact text.\n- [ ] The bootstrapper is idempotent and survives two concurrent instances, using the catch rule from T0 Q7. It\n re-applies a changed TTL. It uses `BootstrapFailureCache` for the 5 s negative cache.\n- [ ] `QuestDbFixture` reads `HISTORIAN_QUESTDB_TEST_CONNECTION` and skips when it is absent. `UniquePrefix()` returns\n `\"t\" + 12 hex + \"_\"`. `DropPrefixAsync(prefix)` drops every table with that prefix, found through `tables()`.\n- [ ] `BootstrapTests` cover:\n - tables created;\n - a second run is a no-op;\n - two concurrent bootstrappers both succeed;\n - TTL is visible in metadata when `RetentionDays=7` and absent when unset;\n - `AutoCreateSchema=false` against a missing table throws `HistorianSchemaMissingException` from Sql.Common.\n\n**Verify:**\n- `dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests` passes.\n- With `HISTORIAN_QUESTDB_TEST_CONNECTION` set, `dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests --filter \"FullyQualifiedName~Bootstrap\"`\n passes.\n- Without the variable, every test skips.\n\n**Steps:**\n- [ ] Step 1: write the options with `Validate()` and its unit tests (red, then green).\n- [ ] Step 2: write `QuestDbTables` DDL builders (use the spec \u00a72.1 *as amended by T0*), with unit tests pinning the\n text for prefix `hist_` with and without TTL.\n- [ ] Step 3: write `QuestDbConnection`.\n - Members:\n - `NpgsqlDataSource` built once;\n - `OpenAsync(ct)`;\n - `CreateCommand(conn, sql)` with `CommandTimeout`;\n - `static bool IsTransient(Exception)`, true for `NpgsqlException` with `IsTransient`, socket and IO errors, and\n timeouts;\n - `static DateTime ReadUtc(NpgsqlDataReader r, int i)`;\n - `string T(string table)`, which returns prefix + table.\n - Use the parameter typing from T0 Q1.\n- [ ] Step 4: write the bootstrapper and fixture. Bring up the overlay\n (`docker compose -f docker-dev/docker-compose.yml -f docker-dev/docker-compose.questdb.yml up -d questdb`), then\n write and run `BootstrapTests`.\n- [ ] Step 5: wire the slnx and the parity-test reference, then build the solution.\n- [ ] Step 6: commit `feat(historian): QuestDB backend skeleton \u2014 options, connection, schema bootstrap with TTL` with\n the trailer.\n\n```json:metadata\n{\"files\":[\"src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/\",\"tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests/\",\"tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/\",\"ZB.MOM.WW.OtOpcUa.slnx\"],\"verifyCommand\":\"dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests --filter FullyQualifiedName~Bootstrap (env set)\",\"acceptanceCriteria\":[\"options validate naming keys\",\"DDL pinned by unit tests\",\"bootstrap idempotent + concurrent + TTL\",\"fixture env-gated with unique prefix\"],\"modelTier\":\"standard\",\"model\":\"sonnet\",\"wave\":2}\n```\n\n---", + "blockedBy": [ + 1 + ] + }, + { + "id": 4, + "subject": "Task 4: QuestDB writes (value writer, tag provisioner, alarm wr", + "status": "pending", + "description": "**Goal:** the recorder, provisioning and alarm-history seams write to QuestDB with the same semantics as SQL Server.\n\n**Files:**\n- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianValueWriter.cs`,\n `QuestDbTagProvisioner.cs`, `QuestDbAlarmHistorianWriter.cs`\n- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/WriteTests.cs`,\n `ProvisioningTests.cs`, `AlarmWriterTests.cs`\n\n**Acceptance Criteria:**\n- [ ] The value writer works as follows:\n - It calls `LiveWritePreparer.Prepare(values, utcNow, resolutionTicks: 10)`.\n - It rejects an invalid tag name (`HistorianTagNames.IsValid`) and returns `false`.\n - It reads the existing timestamps for the tag in the batch range, including tombstones, using the T0 Q11 form.\n - It inserts only the rows that are not yet present, in multi-row parameterised `INSERT`s of at most the T0 Q2\n batch size, with `deleted=false` and `server_ts=utcNow()`.\n - It returns `true` with no SQL when every row already existed.\n - Any exception returns `false`. It never throws, and cancellation also returns `false`.\n- [ ] The provisioner upserts `hist_tag` (`ts = 1970-01-01`, data type from `HistorianDataTypes.FromKind`, eng unit,\n description). Both the request field types and the result tally match the SqlServer provisioner's.\n- [ ] The alarm writer works as follows:\n - It uses the same deterministic `event_id` as the SQL Server writer, through `AlarmEventIds.ComputeEventId` from\n Sql.Common.\n - It inserts events through the dedup key `(ts, event_id)`, so a redelivery is idempotent.\n - `Classify` maps an unknown or non-database exception to RetryPlease, and maps permanent errors as the SqlServer\n writer does.\n- [ ] `WriteTests` cover:\n - typed round trip for all six kinds with a non-Good status;\n - a redelivered batch leaves one row and the first value;\n - a tombstoned key is not resurrected by redelivery (seed a tombstone directly);\n - NaN \u21d2 `BadOutOfRange` Empty;\n - a microsecond collision within a batch keeps the first;\n - a stopped port (`Port=1`) returns `false` without throwing.\n- [ ] Each regression test is falsified once.\n\n**Verify:** with `HISTORIAN_QUESTDB_TEST_CONNECTION` set,\n`dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests --filter \"FullyQualifiedName~Write|FullyQualifiedName~Provisioning|FullyQualifiedName~AlarmWriter\"`\npasses.\n\n**Steps:**\n- [ ] Step 1: write `WriteTests` (red).\n- [ ] Step 2: implement the value writer.\n- [ ] Step 3: make `WriteTests` green, then falsify the first-wins, tombstone and NaN tests.\n- [ ] Step 4: do the provisioner, then the alarm writer, each tests-first.\n- [ ] Step 5: commit `feat(historian): QuestDB live writes, provisioning and alarm events` with the trailer.\n\n```json:metadata\n{\"files\":[\"src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianValueWriter.cs\",\"src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbTagProvisioner.cs\",\"src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbAlarmHistorianWriter.cs\",\"tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/WriteTests.cs\",\"tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/ProvisioningTests.cs\",\"tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/AlarmWriterTests.cs\"],\"verifyCommand\":\"dotnet test \u2026QuestDb.IntegrationTests --filter Write|Provisioning|AlarmWriter (env set)\",\"acceptanceCriteria\":[\"insert-if-absent incl tombstones\",\"never throws\",\"event id deterministic + idempotent\",\"falsified\"],\"modelTier\":\"standard\",\"model\":\"sonnet\",\"wave\":3}\n```\n\n---", + "blockedBy": [ + 3 + ] + }, + { + "id": 5, + "subject": "Task 5: QuestDB reads and aggregates", + "status": "pending", + "description": "**Goal:** `QuestDbHistorianDataSource` serves Raw, AtTime, Events and Processed. Results are identical to SQL Server's,\nexcept for microsecond rounding.\n\n**Files:**\n- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianDataSource.cs`,\n `QuestDbAggregateQueries.cs`\n- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/ReadTests.cs`\n\n**Acceptance Criteria:**\n- [ ] Every query filters `NOT deleted`. Every row goes through `StoredSample.FromStored`.\n- [ ] **Raw:** forward and reverse, `LIMIT n+1`, and the same paging contract and continuation semantics as the\n SqlServer data source (re-read from the last timestamp, inclusive). Unbounded when `maxValuesPerNode == 0`.\n Unknown tag \u21d2 an empty result. The tag is resolved through the `hist_tag` existence or the sample table, so the\n read never creates a tag.\n- [ ] **AtTime:** the result list matches the input in length and order. Neighbours come from the T0 Q9 form, and\n `AtTimeEvaluator.Evaluate` decides each value.\n- [ ] **Processed:**\n - It uses `BucketPlan.Create` and `AggregateFamilies.FamilyOf`.\n - It runs one statement per node, grouped by `(cast(ts AS LONG)*10 - @startUnixTicks) / @intervalTicks`.\n - It fills the shared `BucketRow` with the same usable/good/bad/non-numeric definitions as SQL Server: usable = status\n severity not Bad and kind \u2260 Empty; good = usable and status < 0x40000000; non-numeric = usable and kind in (4, 5).\n - `AggregateEvaluator.Evaluate` produces the values, and `BucketPlan.Assemble` fills empty buckets and handles\n reversal.\n - An unsupported aggregate throws `HistoryAggregateNotSupportedException`.\n- [ ] **Events:** filtered by source and time range, ascending, limited to `maxEvents`. The `ContinuationPoint` is\n non-null when truncated.\n- [ ] Health counters (`GetHealthSnapshot`) are implemented as in the SqlServer data source (lock-guarded success/failure counters plus the last error). A failure fails only that call and is recorded in\n `GetHealthSnapshot`.\n- [ ] `ReadTests` cover only QuestDB-specific behaviour:\n - tombstoned rows are invisible to Raw, AtTime and Processed (seed a tombstone directly);\n - a whole-range Processed read from 1601 to 9999 with interval 0 does not overflow;\n - a reverse paging walk with `maxValuesPerNode=1` returns every row.\n The shared literals run through the conformance suite in T7.\n\n**Verify:** with `HISTORIAN_QUESTDB_TEST_CONNECTION` set,\n`dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests --filter \"FullyQualifiedName~ReadTests\"`\npasses.\n\n**Steps:**\n- [ ] Step 1: write `ReadTests` (red).\n- [ ] Step 2: implement Raw, then AtTime, Events and Processed.\n- [ ] Step 3: temporarily compile the conformance read classes into this task's worktree. Add a throwaway concrete\n subclass and a throwaway backend that seeds through direct INSERTs. Run the Raw, AtTime and Processed\n conformance classes locally and get them green.\n- [ ] Step 4: delete the throwaway files before committing. T7 owns the real ones. Paste the conformance run output\n into the report.\n- [ ] Step 5: falsify the tombstone filter and the bucket expression (an off-by-one start).\n- [ ] Step 6: commit `feat(historian): QuestDB reads and aggregates (pushed down, shared evaluation)` with the trailer.\n\n```json:metadata\n{\"files\":[\"src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianDataSource.cs\",\"src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbAggregateQueries.cs\",\"tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/ReadTests.cs\"],\"verifyCommand\":\"dotnet test \u2026QuestDb.IntegrationTests --filter FullyQualifiedName~ReadTests (env set) + throwaway conformance run in report\",\"acceptanceCriteria\":[\"NOT deleted everywhere\",\"bucket expr exact vs SqlServer\",\"shared evaluators used\",\"conformance read classes green locally\"],\"modelTier\":\"frontier\",\"model\":\"opus\",\"wave\":3}\n```\n\n---", + "blockedBy": [ + 3 + ] + }, + { + "id": 6, + "subject": "Task 6: QuestDB HistoryUpdate (soft delete) with read-your-writ", + "status": "pending", + "description": "**Goal:** `QuestDbHistorianDataUpdater` implements all five operations through insert-only upserts and tombstones. It\nreturns codes identical to SQL Server's, and waits for WAL visibility before returning.\n\n**Files:**\n- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianDataUpdater.cs`,\n `QuestDbWalVisibility.cs`\n- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/UpdateTests.cs`,\n `TombstoneTests.cs`\n\n**Acceptance Criteria:**\n- [ ] `UpdateDataAsync`:\n 1. `HistoryUpdatePlanner.Plan(tag, mode, values, resolutionTicks: 10)` builds the plan.\n 2. It reads which winner timestamps currently have a **live** row, meaning present and not deleted.\n 3. It writes:\n - Insert: only the absent winners;\n - Replace: only the present winners;\n - Update: every winner.\n Each write is an `INSERT` with `deleted=false`.\n 4. It builds the outcome map (present \u21d2 Replaced, absent \u21d2 Inserted, for rows it wrote).\n 5. `HistoryUpdatePlanner.Complete(...)` produces the codes.\n- [ ] `DeleteAtTimeAsync` writes a tombstone for each time with a live row, with `deleted=true`, value columns NULL and\n `status=BadNoData`. Codes come from `HistoryUpdatePlanner.DeleteAtTimeResults`.\n- [ ] `DeleteRawAsync` writes a tombstone for every live row in `[start, end)`. The code comes from\n `HistoryUpdatePlanner.DeleteRawResult`.\n- [ ] On an unknown tag, Insert and Update create the `hist_tag` row; Replace and Delete do not, and return the shared\n unknown-tag codes.\n- [ ] `QuestDbWalVisibility.WaitAsync(table, ct)` works as follows:\n - It polls the T0 Q4 query until the table's applied transaction is at least the sequencer transaction captured\n after our write.\n - It gives up after `VisibilityTimeout` and logs a Warning, without changing the codes. `VisibilityTimeout = 0`\n skips the wait.\n - Every updater operation calls it once, after writing.\n- [ ] `UpdateTests` cover the five operations' codes (QuestDB-specific checks only; the literal codes run through\n conformance in T7) and that a read made right after `UpdateDataAsync` returns sees the change.\n- [ ] `TombstoneTests` cover:\n - delete then Raw read \u21d2 the row is gone;\n - delete, then a `QuestDbHistorianValueWriter` redelivery of the same sample \u21d2 the row stays gone (this needs T4's\n writer: write the test against the interface, and if T4 is not merged yet, seed a tombstone and assert the\n writer's existence query counts it);\n - insert over a tombstone \u21d2 `GoodEntryInserted` and a visible row.\n\n**Verify:** with `HISTORIAN_QUESTDB_TEST_CONNECTION` set,\n`dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests --filter \"FullyQualifiedName~UpdateTests|FullyQualifiedName~TombstoneTests\"`\npasses.\n\n**Steps:**\n- [ ] Step 1: write the tests (red).\n- [ ] Step 2: implement `QuestDbWalVisibility`.\n- [ ] Step 3: implement the updater.\n- [ ] Step 4: get to green. Falsify the \"live row\" filter by dropping `NOT deleted` from the existence check, and the\n visibility wait by setting the timeout to 0 in the read-after-write test.\n- [ ] Step 5: commit `feat(historian): QuestDB HistoryUpdate with soft delete and read-your-writes` with the trailer.\n\n```json:metadata\n{\"files\":[\"src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianDataUpdater.cs\",\"src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbWalVisibility.cs\",\"tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/UpdateTests.cs\",\"tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/TombstoneTests.cs\"],\"verifyCommand\":\"dotnet test \u2026QuestDb.IntegrationTests --filter UpdateTests|TombstoneTests (env set)\",\"acceptanceCriteria\":[\"insert-only upserts, no UPDATE/DELETE\",\"codes from shared planner\",\"tombstones never resurrected\",\"read-your-writes wait\"],\"modelTier\":\"standard\",\"model\":\"sonnet\",\"wave\":3}\n```\n\n---", + "blockedBy": [ + 3 + ] + }, + { + "id": 7, + "subject": "Task 7: Provider, Host wiring and the conformance suite on Ques", + "status": "pending", + "description": "**Goal:** `Backend=QuestDb` resolves, validates and builds all five seams. The full conformance suite passes against\nQuestDB.\n\n**Files:**\n- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianProvider.cs`\n- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/QuestDbConformanceBackend.cs`,\n `QuestDbConformance.cs` (one concrete subclass per conformance class)\n- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests/QuestDbHistorianProviderTests.cs`\n- Modify:\n - `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/HistorianBackendRegistry.cs` (add a `QuestDbBackendName`\n const, `BindQuestDb(IConfiguration)` and a `Factories` entry);\n - `ServerHistorianOptionsValidator.cs` (a QuestDb block before the `if (!options.IsGatewayBackend) return;` line,\n mirroring the SqlServer block);\n - `src/Server/ZB.MOM.WW.OtOpcUa.Host/ZB.MOM.WW.OtOpcUa.Host.csproj` (a ProjectReference);\n - `tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests/Configuration/HistorianBackendRegistryTests.cs`;\n - `tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/ServerHistorianOptionsValidatorTests.cs`;\n - `docker-dev/docker-compose.yml`: in the central-1 and central-2 `ServerHistorian__*` blocks, add\n `ServerHistorian__QuestDb__ConnectionString: \"${OTOPCUA_HISTORIAN_QUESTDB:-}\"`\n plus a comment line naming the overlay.\n\n**Acceptance Criteria:**\n- [ ] The provider has `Name=\"QuestDb\"`, `SupportedKinds` = all six kinds as a FrozenSet, and one lazily built shared\n connection and bootstrapper across the seams. The constructor does no I/O and no validation, like the SqlServer\n provider. `CreateUpdater` returns the soft-delete updater.\n- [ ] The registry resolves `QuestDb` case-insensitively.\n- [ ] The validator, when the backend is consumed and is `QuestDb`:\n - reports every `QuestDbHistorianOptions.Validate()` error, naming the key;\n - does not require the Gateway keys;\n - ignores an invalid `TablePrefix` when not consumed.\n- [ ] Registry and validator tests mirror the five SqlServer ones (lines 71 and 319\u2013368).\n- [ ] The QuestDB integration project runs the **entire** conformance suite through `QuestDbConformanceBackend`:\n - `ResolutionTicks = 10`;\n - seeding is direct INSERTs into the prefixed tables;\n - `ReadBackAsync` filters `NOT deleted`.\n Every class passes. Any named override has a doc comment and a ledger ruling.\n\n**Verify:**\n- With both environment variables set, the SqlServer and QuestDB integration projects both pass. The report shows the\n conformance class counts side by side.\n- `dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests --filter \"FullyQualifiedName~Historian\"` passes.\n- `dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests --filter \"FullyQualifiedName~Historian\"` passes.\n\n**Steps:**\n- [ ] Step 1: write the provider unit tests (red), then the provider.\n- [ ] Step 2: write the registry and validator tests (red), then the wiring.\n- [ ] Step 3: write `QuestDbConformanceBackend` and the subclasses, and run the full suite. Fix any divergence in the\n QuestDB code, even though those files belong to T4\u2013T6; this task is the integration owner in W4.\n- [ ] Step 4: make the compose env edit.\n- [ ] Step 5: build the solution, then commit `feat(historian): QuestDb provider + Host wiring; conformance suite green\n on both backends` with the trailer.\n\n```json:metadata\n{\"files\":[\"src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianProvider.cs\",\"tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/QuestDbConformanceBackend.cs\",\"tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/QuestDbConformance.cs\",\"src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/HistorianBackendRegistry.cs\",\"src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/ServerHistorianOptionsValidator.cs\",\"src/Server/ZB.MOM.WW.OtOpcUa.Host/ZB.MOM.WW.OtOpcUa.Host.csproj\",\"docker-dev/docker-compose.yml\"],\"verifyCommand\":\"both IT projects green with env set; Host historian filters green\",\"acceptanceCriteria\":[\"provider shape\",\"registry + consumer-gated validator\",\"full conformance suite green on QuestDB\",\"compose env\"],\"modelTier\":\"standard\",\"model\":\"sonnet\",\"wave\":4}\n```\n\n---", + "blockedBy": [ + 2, + 4, + 5, + 6 + ] + }, + { + "id": 8, + "subject": "Task 8: Documentation", + "status": "pending", + "description": "**Goal:** the docs describe the QuestDB backend as built, and every claim is checked against the merged code.\n\n**Files:** `docs/Historian.md` (a new \u00a7\"QuestDB backend\"), `docs/Configuration.md` (the `ServerHistorian:QuestDb`\ntable), `CLAUDE.md` (one short bullet in the Pluggable backends subsection), `docker-dev/README.md` (the overlay\nrecipe), and the spec (an as-built pass)\n\n**Acceptance Criteria:**\n- [ ] Every `ServerHistorian:QuestDb` key is documented with its default and validation rule.\n- [ ] The QuestDB-specific behaviour is documented:\n - soft delete and tombstones;\n - insert-if-absent including tombstones;\n - microsecond truncation and collisions;\n - WAL visibility and the read-your-writes wait;\n - TTL;\n - the overlay recipe and `HISTORIAN_QUESTDB_TEST_CONNECTION`.\n- [ ] Each statement is cross-checked against the code, and any spec-versus-code disagreement is fixed in the spec\n inline and listed in the report.\n\n**Verify:** the controller reads the diff, and `grep -n \"QuestDb\" docs/Configuration.md` shows the table.\n\n```json:metadata\n{\"files\":[\"docs/Historian.md\",\"docs/Configuration.md\",\"CLAUDE.md\",\"docker-dev/README.md\",\"docs/plans/2026-10-10-questdb-historian-design.md\"],\"verifyCommand\":\"grep -n QuestDb docs/Configuration.md\",\"acceptanceCriteria\":[\"keys documented\",\"QuestDB-specific semantics documented\",\"claims cross-checked\"],\"modelTier\":\"standard\",\"model\":\"sonnet\",\"wave\":5}\n```\n\n---", + "blockedBy": [ + 7 + ] + }, + { + "id": 9, + "subject": "Task 9: Live gate on docker-dev (controller)", + "status": "pending", + "description": "**Final review:** a **fable** reviewer reads `git diff master...feat/questdb-historian` together with the spec. It must\nreport PASS or FAIL on each of these checks:\n1. SqlServer behaviour is unchanged: identical conformance results, and no deleted SqlServer test.\n2. No QuestDB `UPDATE` or `DELETE` is issued on data.\n3. `NOT deleted` is present in every read path.\n4. Every new status literal is a named const.\n5. No package or version changes.\n6. The bucket expression is exact.\n7. #491 still holds: no Self/Context access after an await.\n\nThen one fix dispatch (opus) and a scoped re-review (sonnet), followed by adjudication.\n\n**Task 9: live gate (controller only).**\n\n**Goal:** prove `Backend=QuestDb` end to end on the docker-dev rig, against the overlay `questdb` service, with captured\noutput for every leg.\n\n> **USER-ORDERED GATE \u2014 NON-SKIPPABLE.** This task was requested by the user in the current conversation. It MUST NOT\n> be closed by walking around it, by declaring it \"verified inline\", or by substituting a cheaper check. Close only\n> after every item in `acceptanceCriteria` has been re-validated independently, with output captured.\n\n**Files:** `docs/plans/2026-10-10-questdb-historian-live-gate.md`\n\n**Acceptance Criteria:** run the six legs of spec \u00a77 on docker-dev with `OTOPCUA_HISTORIAN_BACKEND=QuestDb`. Each leg\nneeds captured output.\n\n**Verify:** the evidence doc has each leg's CLI output plus the direct QuestDB query output.\n\n**Steps:**\n1. Bring up the overlay `questdb`.\n2. Rebuild the central image and recreate the central nodes with `OTOPCUA_HISTORIAN_ENABLED=true`,\n `OTOPCUA_HISTORIAN_BACKEND=QuestDb` and `OTOPCUA_CONTHIST_ENABLED=true`.\n3. Historize FastUInt1, a string calculation tag and `HR5`. Add a UNS reference. Mint the two gate API keys, as in the\n SQL Server gate recipe (`docs/plans/2026-10-09-pluggable-historian-live-gate.md`), and deploy.\n4. Run legs 1\u20136 with direct `psql` or REST queries. QuestDB REST is `curl -G localhost:19000/exec --data-urlencode \"query=\u2026\"`.\n5. Write the evidence doc.\n6. Restore the rig: recreate the centrals with defaults, revert the config DB, redeploy, `down` the overlay, and stop\n the fixtures.\n\n```json:metadata\n{\"files\":[\"docs/plans/2026-10-10-questdb-historian-live-gate.md\"],\"verifyCommand\":\"Client.CLI historyread/historyupdate against opc.tcp://localhost:4840 + QuestDB REST /exec on :19000\",\"acceptanceCriteria\":[\"Primary-only typed recording one row per value\",\"raw+UNS HistoryRead/Processed identical and match QuestDB\",\"HistoryUpdate codes + delete not resurrected by redelivery + WriteOperate-only and anonymous denied\",\"failover 1x\",\"Modbus loss: BadCommunicationError then BadNoCommunication marker then Good\",\"TTL visible in tables() metadata\"],\"userGate\":true,\"tags\":[\"user-gate\"],\"modelTier\":\"frontier\",\"model\":\"controller\",\"wave\":5}\n```\n\n---", + "blockedBy": [ + 8 + ] + } + ], + "lastUpdated": "2026-10-10T12:55:44.872528Z" +} \ No newline at end of file From f54ab04e644836c7018c738c339ff0b64b884476 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Sat, 10 Oct 2026 09:09:42 -0400 Subject: [PATCH 03/23] docs(historian): fold Fable pre-execution review into QuestDB plan + spec (6 blockers, 15 should-fix, 8 nits) Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe --- .../2026-10-10-questdb-historian-design.md | 11 +- .../2026-10-10-questdb-historian-plan.md | 215 +++++++++++++++++- ...10-10-questdb-historian-plan.md.tasks.json | 20 +- 3 files changed, 228 insertions(+), 18 deletions(-) diff --git a/docs/plans/2026-10-10-questdb-historian-design.md b/docs/plans/2026-10-10-questdb-historian-design.md index c01068be..17574b8c 100644 --- a/docs/plans/2026-10-10-questdb-historian-design.md +++ b/docs/plans/2026-10-10-questdb-historian-design.md @@ -28,7 +28,7 @@ Out of scope: This is extracted from `Driver.Historian.SqlServer` with **no behaviour change**: -- **`StoredSample`.** It holds the time, the value, the stored status and a deleted flag. It classifies a sample as +- **`StoredSample`.** It holds the time, the value, the stored status. It classifies a sample as usable, good, bad or uncertain. It applies the quality-contract clamp: a NULL value carrying a non-Bad status is served as `BadNoData`. It also defines the info-bit masks (`CodeBitsMask`, `SeverityBadBit`, `SeverityMask`). - **`AtTimeEvaluator`.** This is the former `SqlServerHistorianDataSource.AtTimeValue`. It covers interpolation and @@ -83,7 +83,7 @@ three are **WAL tables with `DEDUP UPSERT KEYS`**, so re-sending a row with the ```sql CREATE TABLE IF NOT EXISTS hist_sample ( - ts TIMESTAMP, tag SYMBOL, kind BYTE, num DOUBLE, int LONG, str VARCHAR, + ts TIMESTAMP, tag SYMBOL, kind BYTE, num_value DOUBLE, int_value LONG, str_value VARCHAR, status LONG, deleted BOOLEAN, server_ts TIMESTAMP ) TIMESTAMP(ts) PARTITION BY DAY WAL DEDUP UPSERT KEYS(ts, tag) [TTL n DAYS]; @@ -118,8 +118,8 @@ Column details: 5. Any SQL or transport failure makes the write return `false`, so the recorder retries and the #538 per-tag quarantine still applies. It never throws. -The recorder is Primary-only (#537), so step 2 does not race another live writer. The recorder does not wait for -WAL visibility. +The recorder is Primary-only (#537), so step 2 does not race another live writer. The recorder waits for WAL +visibility, bounded by `VisibilityTimeout` (plan R5). ### 2.3 HistoryUpdate (soft delete) @@ -155,8 +155,7 @@ kept forever. `hist_tag` never expires. Every query filters `NOT deleted`. Every value goes through `StoredSample`, so the quality clamp is the same as SQL Server's. -- **Raw.** A range on `ts`, `ORDER BY ts [DESC]`, `LIMIT n+1` to detect another page. The dedup key forbids - same-instant ties, so the continuation is just the last timestamp. Bounds follow the existing data-source contract. +- **Raw.** A range on `ts`, `ORDER BY ts [DESC]`, `LIMIT @top` (`maxValuesPerNode == 0` ⇒ unbounded), at most `top` samples, `ContinuationPoint: null`; the node manager pages by re-reading from the last timestamp (plan R8). The dedup key forbids same-instant ties. Bounds follow the existing data-source contract. - **AtTime.** Find the previous-or-equal neighbour and the next neighbour of each requested time. The default is `LIMIT 1` seeks on `ts`, batched into chunked statements; T0 picks `ASOF JOIN` for the previous neighbour if it measures better. `AtTimeEvaluator` then decides the result. diff --git a/docs/plans/2026-10-10-questdb-historian-plan.md b/docs/plans/2026-10-10-questdb-historian-plan.md index 7bdf912c..4d4039e4 100644 --- a/docs/plans/2026-10-10-questdb-historian-plan.md +++ b/docs/plans/2026-10-10-questdb-historian-plan.md @@ -91,6 +91,67 @@ These are binding on every task and every reviewer. They are renamed from their `Sql*` names to neutral ones (table in T1). The parity test keys on field names, not class names. +## Review amendments (Fable, 2026-10-10) + +These are binding. They override anything else in this plan that conflicts with them. The full review is +summarised here as B = blocker, S = should-fix, N = nit. + +**Global rulings** +- **R4 — resolution units (B2).** Conformance tests express every sub-second offset in units of `B.ResolutionTicks`: + - `AddTicks(k)` becomes `AddTicks(k * B.ResolutionTicks)`. + - Seeds are at least one resolution unit apart. + - A "near miss" is `+ B.ResolutionTicks`, and the last tick of a bucket is `edge.AddTicks(-B.ResolutionTicks)`. + - The 100k-bucket cap test uses `TimeSpan.FromTicks(B.ResolutionTicks)` intervals over a + `100_000 * B.ResolutionTicks` window. + - Every timestamp assertion, including `.Ticks` comparisons and `ServerTimestampUtc`, goes through `B.Round(..)`. + - Fixture bases (`T0`, `T(n)`) stay deliberately not µs-aligned. + - On SQL Server (`ResolutionTicks = 1`) every rewritten expression evaluates to today's literal. +- **R5 — both write paths wait for WAL visibility (B6).** `QuestDbWalVisibility` is a T3 file. The value writer + (T4) and the updater (T6) each call `WaitAsync` once after writing, bounded by `VisibilityTimeout`. A timeout logs a + Warning and does not change the return value or result codes. Spec §2.2's "The recorder does not wait for WAL + visibility" is replaced by "The recorder waits for WAL visibility, bounded by `VisibilityTimeout`." +- **R6 — microsecond domain at the boundary (B3, B4).** + - `QuestDbConnection.Round(DateTime)` (= `ticks - ticks % 10`, UTC kind) is applied to EVERY inbound timestamp: + - Raw and Events window bounds; + - each AtTime request time; + - DeleteRaw bounds and DeleteAtTime times; + - the Processed window, BEFORE `BucketPlan.Create`. + - Results are stamped with the rounded request time. + - `QuestDbConnection.BindTimestamp(cmd, name, DateTime)` is the ONLY way a DateTime reaches Npgsql. It rounds, and it + never sends a sentinel: `DateTime.MinValue` becomes `MinValue.AddTicks(10)`, and `MaxValue` rounds to `MaxValue − 9` + ticks. Binding a raw DateTime is a review FAIL. + - The Global Constraints "Bucket edges" rule therefore holds exactly. T8 documents that a request start with sub-µs + ticks admits a sample stored up to 9 ticks earlier. +- **R7 — one tag authority (B5).** T3 creates `QuestDbTagCatalog.cs`, mirroring `SqlServerTagCatalog`: + - `Task ExistsAsync(name, ct)`: positive results cached; an invalid name ⇒ false; runs bootstrap `EnsureAsync` + first. + - `Task EnsureAsync(name, byte dataType, ct)`: inserts into `hist_tag` with `ts = 1970-01-01` when absent. + - **The `hist_tag` row is the authority for "known tag".** T4's writer and T6's Insert/Update call `EnsureAsync`. + T5's reads and T6's Replace/DeleteRaw/DeleteAtTime call `ExistsAsync` and never create. + - T4–T6 add no private tag resolver. +- **R8 — Raw paging (S1).** `LIMIT @top`, where `top = maxValuesPerNode == 0 ? int.MaxValue : maxValuesPerNode`. Return + at most `top` samples with `ContinuationPoint: null`, exactly as the SqlServer data source does. The node manager does + the paging (it re-reads from the last timestamp, inclusive). The spec §3 "LIMIT n+1" is wrong and is replaced. +- **R9 — value columns (S3).** In the QuestDB DDL the value columns are `num_value`, `int_value` and `str_value`, to + avoid the reserved word `int`. The `SampleColumns` names in Sql.Common are unchanged. +- **R10 — models (S9, S10).** + - T7 runs on **opus**. It may fix T4/T6 files and the Raw/Events plumbing in `QuestDbHistorianDataSource.cs`. + - A divergence inside `QuestDbAggregateQueries.cs` or the AtTime neighbour logic is reported, not fixed; the + controller dispatches an opus fix. + - T1 runs as **two sequential opus dispatches**: + - **1a:** extract, SqlServer re-point, rename fixes, slnx and parity wiring, moved unit tests. Its gate is the Step 0 + counts, reproduced exactly. + - **1b:** the new direct `Sql.Common.Tests`, each falsified. + - Two commits, one review each. +- **R11 — ownership (B1, N6, S8).** + - `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/**`: T1 in W1, for **mechanical + rename and using fixes only** (no test moved, renamed, re-literalled or deleted; the count stays at baseline), then + T2 in W2. + - `IT/QuestDbConformanceBackend.cs` (seed and read-back only) is owned by **T5**. T7 owns the subclasses + (`QuestDbConformance.cs`). + - `IT/TombstoneTests.Redelivery.cs` is owned by T7 (see the T6 amendment). +- **N4.** `Platforms AnyCPU;x64` goes on both new src csprojs. + --- ## Waves, models and parallel safety @@ -101,7 +162,7 @@ These are binding on every task and every reviewer. | W1 | T1 extract `Sql.Common` | **opus** | | W2 | T2 conformance suite over SqlServer ‖ T3 QuestDB skeleton | sonnet ‖ sonnet | | W3 | T4 writes ‖ T5 reads + aggregates ‖ T6 updater | sonnet ‖ **opus** ‖ sonnet | -| W4 | T7 provider + Host wiring + QuestDB conformance | sonnet | +| W4 | T7 provider + Host wiring + QuestDB conformance | **opus** (R10) | | W5 | T8 docs → **fable** whole-branch review → one fix wave (**opus**) + scoped re-review (sonnet) → T9 live gate | sonnet / fable / opus / controller | Reviewers: @@ -140,6 +201,33 @@ T3 already wired. ### Task 0: QuestDB spike, overlay compose and spec amendments +> **Review amendments (Fable) — binding:** +> - **Q1 is expanded (S2).** Record, with output: +> - the DateTime `Kind`/`NpgsqlDbType` pair that writes `TIMESTAMP` without error (watch for +> `Cannot write DateTime with Kind=UTC to … timestamp without time zone`); +> - whether open → close → reopen on one `NpgsqlDataSource` works without `No Reset On Close=true` (DISCARD ALL); +> - the CLR type returned for `BYTE` (likely `int2`, so use `GetInt16`), `SYMBOL`, `VARCHAR`, `LONG`, `BOOLEAN` and +> `TIMESTAMP`; +> - the parameter typing needed for `$n` in integer arithmetic (`NpgsqlDbType.Bigint`); +> - that a DOUBLE NaN reads as NULL. +> The confirmed keywords go into spec §4 verbatim. +> - **Q3 (S4):** also test `hist_event` with `event_id VARCHAR` as a dedup key. **Fallback:** no dedup on `hist_event`. +> The alarm writer checks `SELECT 1 … WHERE event_id=$1` before inserting, and `ReadEventsAsync` de-duplicates by +> `event_id` client-side. +> - **Q4 fallback (S11):** if `wal_tables()` lacks `writerTxn`/`sequencerTxn`, the visibility wait polls for the written +> key (`WHERE tag=$1 AND ts=$2 AND server_ts=$3`). Record which form is used. +> - **Q5 (S11):** the image must be ≥ 8.2.3, the first release with TTL. Record the floor. +> - **Q7 fallback (S11):** catch a `PostgresException` whose message contains `already exists`, then re-check +> `tables()`. Never swallow blindly. +> - **Q8 fallback (S11):** if neither a window `ORDER BY num` nor the `GROUP BY` expression works, arg-min/arg-max is a +> second statement per node, joined client-side. One-query-per-node is relaxed to two for the ActualTime family only. +> - **Q13 (B4):** bind `DateTime.MinValue`, `DateTime.MaxValue`, `0001-01-01`, `1601-01-01`, +> `9999-12-31T23:59:59.9999999`, and a pre-2000 timestamp with sub-µs ticks, in `WHERE ts >= $1 AND ts < $2`. Record +> what arrives and what reads back. This confirms R6's `BindTimestamp` rule. +> - **Healthcheck (S14):** check whether `curl` exists in the image. If not, use `wget -qO-`, or drop the healthcheck. +> - **Q2 also fixes the page size** that T6 uses for DeleteRaw. + + **Goal:** prove every QuestDB behaviour the design assumes, against a pinned image, before any code is built on it. The findings are written into spec §2–§4. @@ -240,6 +328,24 @@ Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe" ### Task 1: Extract `Driver.Historian.Sql.Common` (no behaviour change) +> **Review amendments (Fable) — binding:** +> - Runs as dispatch **1a** then **1b** (R10). +> - **Files gain** `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/**`, for mechanical +> rename and using fixes only (R11 / B1). This includes replacing `SqlServerSchemaBootstrapper.FailureCacheDuration` +> with the `BootstrapFailureCache` constant. +> - `ProcessedReadTests.Every_enum_member_is_either_served_or_refused` moves to `Sql.Common.Tests` (B2 list). +> - **Step 1 API additions (S6):** +> - `AlarmEventIds.ToNumericSeverity(AlarmSeverity)` (moved from the private SqlServer method); +> - `public sealed class HistorianHealthCounters` (lock-guarded): `RecordSuccess()`, `RecordFailure(Exception)` +> (`OperationCanceledException` is not a failure), and `Snapshot()` returning `HistorianHealthSnapshot` with the +> `ProcessConnectionOpen = successes > 0 && consecutive == 0` rule. `SqlServerHistorianDataSource` uses it too. +> - `public static class EventContinuation { string Encode(DateTime, string eventId) }`, in the existing +> `"{ticks}|{EventId}"` format. +> - `HistoricalEventRows.ClampSeverity(int) → ushort`. +> - `StatusBits.SeverityUncertainBit = 0x40000000u`. +> - Copy `Platforms AnyCPU;x64` into the new csproj (N4). + + **Goal:** move every database-independent piece of `Driver.Historian.SqlServer` into a new shared library. The SQL Server backend then calls it, with byte-identical behaviour. @@ -418,6 +524,40 @@ public sealed class BootstrapFailureCache // the 5 s negative-cache mechanism ### Task 2: Conformance suite (shared source), run against SQL Server +> **Review amendments (Fable) — binding:** +> - **AC bullet 1 becomes:** "Every moved test keeps its method name and its expected VALUES/STATUS literals. Tick +> offsets are rewritten per R4. The report lists each test whose offsets were rewritten." +> - **AC bullet 5 becomes:** "A `virtual` override is allowed only for a difference R4 cannot express. The report names +> it, and the controller rules on it before T7." +> - **Step 2 explicit list (B2). Move:** +> - `RawReadTests` — all except `A_missing_schema_surfaces_in_health_LastError_and_is_not_latched`. The SQL tag-count +> line in `Unknown_tag_is_an_empty_result_and_reads_never_create_tags` becomes `B.TagExistsAsync`. +> - `AtTimeReadTests` — all. +> - `ProcessedReadTests` — all except `Every_enum_member_is_either_served_or_refused`, which went to Sql.Common.Tests +> in T1. +> - `QualityContractTests` — all. Direct status-column reads become `B.ReadBackAsync` status checks. +> - `WriteTests` — `Every_kind_round_trips…`, `Writing_the_same_batch_twice_is_idempotent`, +> `Within_batch_duplicate_timestamps…`, `Non_finite_doubles…`, `Empty_batch_succeeds…` and `Tag_is_created…`. +> - `UpdateTests` — all. +> - `LongNameTests` — all except `An_event_read_filtered_by_a_long_source_matches_nothing`. +> - **Stay in SqlServer:** +> - `A_missing_schema…` +> - `Ten_thousand_values_write_in_under_five_seconds` +> - `Concurrent_overlapping_writers…` +> - `A_SQL_failure_returns_false…` +> - `EventReadTests` +> - `ProvisioningTests` +> - `AlarmWriterTests` +> - `SchemaBootstrapIntegrationTests` +> - **Interface (S13):** add `Task TagExistsAsync(string tag);`. `ReadBackAsync` returns the stored status +> verbatim, with no read-side clamp; say so in its doc comment. +> - **Skip check (S7):** first run with the env var UNSET and confirm the tests are reported Skipped, not Failed. +> Fallback: `ConformanceBase` exposes a lazy, memoised `protected async Task BackendAsync()`, +> disposed in `DisposeAsync`, and each test starts with `var B = await BackendAsync();`. +> - **Verify (N8):** the report lists every moved method name, plus per-file `[Fact]`/`[Theory]` counts before and +> after. + + **Goal:** one backend-neutral suite of hand-computed behaviour tests, compiled into the SqlServer integration project and passing there. T7 later compiles the same suite into the QuestDB project. @@ -492,6 +632,19 @@ public abstract class ConformanceBase : IAsyncLifetime ### Task 3: QuestDB skeleton (options, connection, bootstrapper, fixture) +> **Review amendments (Fable) — binding:** +> - **Files gain** `QuestDbTagCatalog.cs` (R7) and `QuestDbWalVisibility.cs` (R5, moved from T6). +> - `QuestDbConnection` gains `Round(DateTime)` and `BindTimestamp` (R6), plus `BindByte` and `ReadKind`, so the T0 Q1 +> answers live in one file (S2). +> - The DDL uses `num_value`/`int_value`/`str_value` (R9). +> - **New tests:** +> - `BootstrapTests.ExistsAsync_is_false_for_an_unknown_tag_and_true_after_EnsureAsync`; +> - `Wait_returns_once_writerTxn_reaches_the_captured_sequencerTxn` (or the Q4 fallback form); +> - `Timeout_zero_skips_the_wait`; +> - a `QuestDb.Tests` unit test pinning that `BindTimestamp` never emits `DateTime.MinValue`/`MaxValue`. +> - Copy `Platforms AnyCPU;x64` (N4). + + **Goal:** a building `Driver.Historian.QuestDb` project. It has validated options, a pooled Npgsql connection, an idempotent and concurrency-safe schema bootstrapper with TTL, and an env-gated integration fixture. @@ -559,6 +712,16 @@ idempotent and concurrency-safe schema bootstrapper with TTL, and an env-gated i ### Task 4: QuestDB writes (value writer, tag provisioner, alarm writer) +> **Review amendments (Fable) — binding:** +> - The writer calls `QuestDbTagCatalog.EnsureAsync` before inserting (R7). After inserting, it calls +> `QuestDbWalVisibility.WaitAsync`; a timeout does not change the return value (R5). +> - **Provisioner (S5):** read the existing `hist_tag` rows per chunk (`WHERE name IN (…)`), merge +> `COALESCE(request, stored)` for `eng_unit` and `description` client-side, then insert. Pin this with the SqlServer +> `Upsert_creates_then_changes_DataType_on_rerun…` literal. +> - The alarm writer uses `AlarmEventIds.ToNumericSeverity`, plus the Q3 fallback if VARCHAR dedup was refused. +> - Every DateTime is bound via `BindTimestamp` (R6). + + **Goal:** the recorder, provisioning and alarm-history seams write to QuestDB with the same semantics as SQL Server. **Files:** @@ -612,6 +775,22 @@ passes. ### Task 5: QuestDB reads and aggregates +> **Review amendments (Fable) — binding:** +> - **Raw per R8:** `LIMIT @top`, at most `top` samples, `ContinuationPoint: null`. Delete the "LIMIT n+1" text. +> - Every inbound timestamp is rounded via `QuestDbConnection.Round`, and every DateTime is bound via `BindTimestamp` +> (R6). The Processed window is rounded before `BucketPlan.Create`. +> - Tags are resolved through `QuestDbTagCatalog.ExistsAsync` only (R7). +> - **SQL classification (S6):** usable ⇔ `status < {StatusBits.SeverityBadBit}` AND `kind <> 0`; good ⇔ usable AND +> `status < {StatusBits.SeverityUncertainBit}`. The literals are interpolated from the consts, never typed by hand. Note +> that QuestDB's "not Bad" is an unsigned LONG comparison, not T-SQL's `>= 0`. +> - Health uses the shared `HistorianHealthCounters`. Events use `EventContinuation` and +> `HistoricalEventRows.ClampSeverity`. +> - **T5 owns `IT/QuestDbConformanceBackend.cs`** (seed and read-back only; N6). Step 3's throwaway backend IS this file, +> which is kept. Only the throwaway subclasses are deleted. +> - **Extra test (N2):** `ReadTests` includes one seeded Int64-kind row with `num_value = NULL`, proving the kind, not +> the column, decides the value. + + **Goal:** `QuestDbHistorianDataSource` serves Raw, AtTime, Events and Processed. Results are identical to SQL Server's, except for microsecond rounding. @@ -669,6 +848,18 @@ passes. ### Task 6: QuestDB HistoryUpdate (soft delete) with read-your-writes +> **Review amendments (Fable) — binding:** +> - `QuestDbWalVisibility.cs` moves to T3, so T6 consumes it (R5). +> - Insert and Update call `QuestDbTagCatalog.EnsureAsync`; Replace and the deletes call `ExistsAsync` (R7). +> - **Existence read (S15):** one `WHERE tag=$1 AND ts >= $min AND ts <= $max AND NOT deleted` range query, filtered to +> the winners client-side. Never an `IN` list of `n` parameters. +> - **DeleteRaw (S12):** read live rows in `LIMIT` pages of the Q2 batch size, keyed on the last `ts`, and tombstone them +> page by page. `rowsAffected` is the total. +> - Inbound times are rounded and bound via `BindTimestamp` (R6). +> - The TombstoneTests "redelivery after delete" case moves to T7 as +> `IT/TombstoneTests.Redelivery.cs::Tombstone_is_not_resurrected_by_a_writer_redelivery` (S8). T6 keeps the other two. + + **Goal:** `QuestDbHistorianDataUpdater` implements all five operations through insert-only upserts and tombstones. It returns codes identical to SQL Server's, and waits for WAL visibility before returning. @@ -730,6 +921,15 @@ passes. ### Task 7: Provider, Host wiring and the conformance suite on QuestDB +> **Review amendments (Fable) — binding:** +> - The model is **opus** (R10). T7 may fix T4/T6 files and the data-source Raw/Events plumbing. Aggregate and AtTime +> divergences are reported to the controller, not fixed. +> - T5 owns `QuestDbConformanceBackend.cs`; T7 owns the subclasses (N6). +> - T7 adds `IT/TombstoneTests.Redelivery.cs` (S8). +> - **Compose default (S14):** the exact Q1-confirmed connection string, with `Host=questdb;Port=8812` substituted, +> keeping every keyword T0 found necessary. + + **Goal:** `Backend=QuestDb` resolves, validates and builds all five seams. The full conformance suite passes against QuestDB. @@ -782,13 +982,24 @@ QuestDB. on both backends` with the trailer. ```json:metadata -{"files":["src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianProvider.cs","tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/QuestDbConformanceBackend.cs","tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/QuestDbConformance.cs","src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/HistorianBackendRegistry.cs","src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/ServerHistorianOptionsValidator.cs","src/Server/ZB.MOM.WW.OtOpcUa.Host/ZB.MOM.WW.OtOpcUa.Host.csproj","docker-dev/docker-compose.yml"],"verifyCommand":"both IT projects green with env set; Host historian filters green","acceptanceCriteria":["provider shape","registry + consumer-gated validator","full conformance suite green on QuestDB","compose env"],"modelTier":"standard","model":"sonnet","wave":4} +{"files":["src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianProvider.cs","tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/QuestDbConformanceBackend.cs","tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/QuestDbConformance.cs","src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/HistorianBackendRegistry.cs","src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/ServerHistorianOptionsValidator.cs","src/Server/ZB.MOM.WW.OtOpcUa.Host/ZB.MOM.WW.OtOpcUa.Host.csproj","docker-dev/docker-compose.yml"],"verifyCommand":"both IT projects green with env set; Host historian filters green","acceptanceCriteria":["provider shape","registry + consumer-gated validator","full conformance suite green on QuestDB","compose env"],"modelTier":"frontier","model":"opus","wave":4} ``` --- ### Task 8: Documentation +> **Review amendments (Fable) — binding:** +> - Document these, in addition to the existing list: +> - R6's sub-µs request-start consequence; +> - QuestDB `LONG` NULL = `long.MinValue`, so a stored `FromInt64(long.MinValue)` reads back Empty (N1); +> - TTL is enforced on write commits only, so a quiet table never drops (N3); +> - all write paths wait for WAL visibility (R5). +> - **CLAUDE.md (N5):** check the Historian section's opening paragraph. If it still says +> "HistorianGateway (sole historian backend)", rewrite it to name all three backends. Otherwise add QuestDb to the +> existing "Pluggable backends" subsection. + + **Goal:** the docs describe the QuestDB backend as built, and every claim is checked against the merged code. **Files:** `docs/Historian.md` (a new §"QuestDB backend"), `docs/Configuration.md` (the `ServerHistorian:QuestDb` diff --git a/docs/plans/2026-10-10-questdb-historian-plan.md.tasks.json b/docs/plans/2026-10-10-questdb-historian-plan.md.tasks.json index 5d35edc6..986a4f8e 100644 --- a/docs/plans/2026-10-10-questdb-historian-plan.md.tasks.json +++ b/docs/plans/2026-10-10-questdb-historian-plan.md.tasks.json @@ -5,13 +5,13 @@ "id": 0, "subject": "Task 0: QuestDB spike, overlay compose and spec amendments", "status": "pending", - "description": "**Goal:** prove every QuestDB behaviour the design assumes, against a pinned image, before any code is built on it.\nThe findings are written into spec \u00a72\u2013\u00a74.\n\n**Files:**\n- Create: `docker-dev/docker-compose.questdb.yml`\n- Create: `docs/plans/2026-10-10-questdb-spike.md`\n- Modify: `docs/plans/2026-10-10-questdb-historian-design.md` (\u00a72\u2013\u00a74: replace assumptions with confirmed forms, and mark\n each one \"T0-confirmed\")\n\n**Acceptance Criteria:**\n- [ ] The overlay pins an exact `questdb/questdb:` tag. The tag's release date is at least two weeks before\n 2026-10-10, and is recorded in the spike doc.\n- [ ] The overlay exposes `18812:8812` and `19000:9000`, uses a named volume `otopcua-questdb-data`, and sets the\n container name `otopcua-dev-questdb-1`. It is brought up with\n `docker compose -f docker-dev/docker-compose.yml -f docker-dev/docker-compose.questdb.yml up -d questdb`.\n- [ ] The spike doc records a working Npgsql connection string and answers Q1\u2013Q12 below, each with the exact SQL or\n C# that worked and its observed output.\n- [ ] Spec \u00a72.1 has the confirmed DDL, \u00a72.3 the confirmed visibility-wait query, \u00a73 the confirmed bucket-statistics and\n neighbour query forms, and \u00a74 the confirmed connection-string keywords.\n\n**Verify:** `docker exec otopcua-dev-questdb-1 true` exits 0. The spike harness output is pasted into the spike doc.\n\n**Steps:**\n\n- [ ] **Step 1: Create the overlay.**\n\n```yaml\n# docker-dev/docker-compose.questdb.yml \u2014 optional QuestDB fixture for the QuestDB historian backend.\n# Up: docker compose -f docker-dev/docker-compose.yml -f docker-dev/docker-compose.questdb.yml up -d questdb\nservices:\n questdb:\n image: questdb/questdb: # T0 pins an exact tag \u2265 2 weeks old\n container_name: otopcua-dev-questdb-1\n ports:\n - \"18812:8812\" # PG wire\n - \"19000:9000\" # web console / REST\n volumes:\n - otopcua-questdb-data:/var/lib/questdb\n healthcheck:\n test: [\"CMD-SHELL\", \"curl -fs http://localhost:9003/status || exit 1\"]\n interval: 10s\n timeout: 5s\n retries: 12\nvolumes:\n otopcua-questdb-data:\n```\n\n- [ ] **Step 2: Write a throwaway C# harness** in the scratchpad, not in the repo. It is a console project that\n references `Npgsql`, restored from the existing central pin via a temporary `Directory.Packages.props` copy, or\n run with `dotnet run` inside a scratch folder under the repo with `ManagePackageVersionsCentrally`. Answer each\n question with real output:\n - **Q1.** The connection-string keywords that work: test `ServerCompatibilityMode=NoTypeLoading` with and without\n it. Also the parameter types needed for `TIMESTAMP` (UTC `DateTime`), `BYTE`, `SYMBOL`, `VARCHAR`, `BOOLEAN` and\n `LONG`.\n - **Q2.** Whether a parameterised multi-row `INSERT INTO t VALUES (\u2026),(\u2026)` works, and the largest row count per\n statement that works reliably. Pick the batch size: \u2264 1000 rows, capped by any parameter limit found.\n - **Q3.** `CREATE TABLE \u2026 WAL DEDUP UPSERT KEYS(ts, tag)`: re-inserting the same key replaces the row. Show\n before and after.\n - **Q4.** The `wal_tables()` columns. After an insert, how to know the transaction we produced and wait until it is\n applied (`writerTxn >= sequencerTxn`). Measure the typical lag over 20 runs.\n - **Q5.** TTL: the syntax at create (`TTL n DAYS`), and `ALTER TABLE \u2026 SET TTL n DAYS`. Where TTL shows up in\n `tables()` metadata.\n - **Q6.** Timestamp precision through Npgsql: a value with 100 ns ticks (`\u20261234567`) comes back as `\u2026123456` (truncated)\n or `\u2026123457` (rounded)? Confirm that our pre-truncation makes the round trip exact.\n - **Q7.** Two connections run `CREATE TABLE IF NOT EXISTS` for the same table at once, 20 times. Record any error\n text, and the catch rule that makes it idempotent.\n - **Q8.** Bucket statistics: one `GROUP BY` over the expression `(cast(ts AS LONG)*10 - $1) / $2`, with\n `sum(CASE WHEN \u2026 THEN 1 ELSE 0 END)`, `avg`, `min`, `max`, `sum`. Plus the first and last usable sample per bucket,\n with its value and time. Try `first()`/`last()` with a filter, or a window `row_number() OVER (PARTITION BY b\n ORDER BY ts)`. Plus arg-min/arg-max time, via a window function or a second statement.\n - **Q9.** AtTime neighbour lookup: chunked `LIMIT 1` seeks (`ts <= t ORDER BY ts DESC LIMIT 1` and\n `ts > t ORDER BY ts LIMIT 1`, combined with `UNION ALL` per time) versus `ASOF JOIN`. Time both for 100 timestamps\n and pick one.\n - **Q10.** VARCHAR and SYMBOL accept a 400-character string, and `SYMBOL` capacity is fine for tens of thousands of\n tags.\n - **Q11.** An existing-timestamp lookup for a batch: `WHERE tag=$1 AND ts >= $2 AND ts <= $3` (a range), or an `IN`\n list. Pick the one that works with parameters.\n - **Q12.** `DROP TABLE IF EXISTS` works for test cleanup, and `tables()` lists the tables with a given prefix.\n\n- [ ] **Step 3: Write the spike doc** with the image tag and date, the connection string, and Q1\u2013Q12 (SQL, output and\n decision for each).\n- [ ] **Step 4: Amend the spec \u00a72\u2013\u00a74** in place with the confirmed forms. If any assumption failed, write the\n replacement design inline and add a ledger ruling. If the failure breaks a user decision, stop and ask.\n- [ ] **Step 5: Commit**\n\n```bash\ngit add docker-dev/docker-compose.questdb.yml docs/plans/2026-10-10-questdb-spike.md docs/plans/2026-10-10-questdb-historian-design.md\ngit commit -m \"spike(historian): QuestDB behaviours confirmed against pinned image; spec \u00a72\u2013\u00a74 amended\n\nClaude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe\"\n```\n\n```json:metadata\n{\"files\":[\"docker-dev/docker-compose.questdb.yml\",\"docs/plans/2026-10-10-questdb-spike.md\",\"docs/plans/2026-10-10-questdb-historian-design.md\"],\"verifyCommand\":\"docker exec otopcua-dev-questdb-1 true\",\"acceptanceCriteria\":[\"pinned image \u22652 weeks old\",\"overlay ports 18812/19000\",\"Q1\u2013Q12 answered with output\",\"spec \u00a72\u2013\u00a74 amended\"],\"modelTier\":\"frontier\",\"model\":\"controller\",\"wave\":0}\n```\n\n---" + "description": "> **Review amendments (Fable) \u2014 binding:**\n> - **Q1 is expanded (S2).** Record, with output:\n> - the DateTime `Kind`/`NpgsqlDbType` pair that writes `TIMESTAMP` without error (watch for\n> `Cannot write DateTime with Kind=UTC to \u2026 timestamp without time zone`);\n> - whether open \u2192 close \u2192 reopen on one `NpgsqlDataSource` works without `No Reset On Close=true` (DISCARD ALL);\n> - the CLR type returned for `BYTE` (likely `int2`, so use `GetInt16`), `SYMBOL`, `VARCHAR`, `LONG`, `BOOLEAN` and\n> `TIMESTAMP`;\n> - the parameter typing needed for `$n` in integer arithmetic (`NpgsqlDbType.Bigint`);\n> - that a DOUBLE NaN reads as NULL.\n> The confirmed keywords go into spec \u00a74 verbatim.\n> - **Q3 (S4):** also test `hist_event` with `event_id VARCHAR` as a dedup key. **Fallback:** no dedup on `hist_event`.\n> The alarm writer checks `SELECT 1 \u2026 WHERE event_id=$1` before inserting, and `ReadEventsAsync` de-duplicates by\n> `event_id` client-side.\n> - **Q4 fallback (S11):** if `wal_tables()` lacks `writerTxn`/`sequencerTxn`, the visibility wait polls for the written\n> key (`WHERE tag=$1 AND ts=$2 AND server_ts=$3`). Record which form is used.\n> - **Q5 (S11):** the image must be \u2265 8.2.3, the first release with TTL. Record the floor.\n> - **Q7 fallback (S11):** catch a `PostgresException` whose message contains `already exists`, then re-check\n> `tables()`. Never swallow blindly.\n> - **Q8 fallback (S11):** if neither a window `ORDER BY num` nor the `GROUP BY` expression works, arg-min/arg-max is a\n> second statement per node, joined client-side. One-query-per-node is relaxed to two for the ActualTime family only.\n> - **Q13 (B4):** bind `DateTime.MinValue`, `DateTime.MaxValue`, `0001-01-01`, `1601-01-01`,\n> `9999-12-31T23:59:59.9999999`, and a pre-2000 timestamp with sub-\u00b5s ticks, in `WHERE ts >= $1 AND ts < $2`. Record\n> what arrives and what reads back. This confirms R6's `BindTimestamp` rule.\n> - **Healthcheck (S14):** check whether `curl` exists in the image. If not, use `wget -qO-`, or drop the healthcheck.\n> - **Q2 also fixes the page size** that T6 uses for DeleteRaw.\n\n\n**Goal:** prove every QuestDB behaviour the design assumes, against a pinned image, before any code is built on it.\nThe findings are written into spec \u00a72\u2013\u00a74.\n\n**Files:**\n- Create: `docker-dev/docker-compose.questdb.yml`\n- Create: `docs/plans/2026-10-10-questdb-spike.md`\n- Modify: `docs/plans/2026-10-10-questdb-historian-design.md` (\u00a72\u2013\u00a74: replace assumptions with confirmed forms, and mark\n each one \"T0-confirmed\")\n\n**Acceptance Criteria:**\n- [ ] The overlay pins an exact `questdb/questdb:` tag. The tag's release date is at least two weeks before\n 2026-10-10, and is recorded in the spike doc.\n- [ ] The overlay exposes `18812:8812` and `19000:9000`, uses a named volume `otopcua-questdb-data`, and sets the\n container name `otopcua-dev-questdb-1`. It is brought up with\n `docker compose -f docker-dev/docker-compose.yml -f docker-dev/docker-compose.questdb.yml up -d questdb`.\n- [ ] The spike doc records a working Npgsql connection string and answers Q1\u2013Q12 below, each with the exact SQL or\n C# that worked and its observed output.\n- [ ] Spec \u00a72.1 has the confirmed DDL, \u00a72.3 the confirmed visibility-wait query, \u00a73 the confirmed bucket-statistics and\n neighbour query forms, and \u00a74 the confirmed connection-string keywords.\n\n**Verify:** `docker exec otopcua-dev-questdb-1 true` exits 0. The spike harness output is pasted into the spike doc.\n\n**Steps:**\n\n- [ ] **Step 1: Create the overlay.**\n\n```yaml\n# docker-dev/docker-compose.questdb.yml \u2014 optional QuestDB fixture for the QuestDB historian backend.\n# Up: docker compose -f docker-dev/docker-compose.yml -f docker-dev/docker-compose.questdb.yml up -d questdb\nservices:\n questdb:\n image: questdb/questdb: # T0 pins an exact tag \u2265 2 weeks old\n container_name: otopcua-dev-questdb-1\n ports:\n - \"18812:8812\" # PG wire\n - \"19000:9000\" # web console / REST\n volumes:\n - otopcua-questdb-data:/var/lib/questdb\n healthcheck:\n test: [\"CMD-SHELL\", \"curl -fs http://localhost:9003/status || exit 1\"]\n interval: 10s\n timeout: 5s\n retries: 12\nvolumes:\n otopcua-questdb-data:\n```\n\n- [ ] **Step 2: Write a throwaway C# harness** in the scratchpad, not in the repo. It is a console project that\n references `Npgsql`, restored from the existing central pin via a temporary `Directory.Packages.props` copy, or\n run with `dotnet run` inside a scratch folder under the repo with `ManagePackageVersionsCentrally`. Answer each\n question with real output:\n - **Q1.** The connection-string keywords that work: test `ServerCompatibilityMode=NoTypeLoading` with and without\n it. Also the parameter types needed for `TIMESTAMP` (UTC `DateTime`), `BYTE`, `SYMBOL`, `VARCHAR`, `BOOLEAN` and\n `LONG`.\n - **Q2.** Whether a parameterised multi-row `INSERT INTO t VALUES (\u2026),(\u2026)` works, and the largest row count per\n statement that works reliably. Pick the batch size: \u2264 1000 rows, capped by any parameter limit found.\n - **Q3.** `CREATE TABLE \u2026 WAL DEDUP UPSERT KEYS(ts, tag)`: re-inserting the same key replaces the row. Show\n before and after.\n - **Q4.** The `wal_tables()` columns. After an insert, how to know the transaction we produced and wait until it is\n applied (`writerTxn >= sequencerTxn`). Measure the typical lag over 20 runs.\n - **Q5.** TTL: the syntax at create (`TTL n DAYS`), and `ALTER TABLE \u2026 SET TTL n DAYS`. Where TTL shows up in\n `tables()` metadata.\n - **Q6.** Timestamp precision through Npgsql: a value with 100 ns ticks (`\u20261234567`) comes back as `\u2026123456` (truncated)\n or `\u2026123457` (rounded)? Confirm that our pre-truncation makes the round trip exact.\n - **Q7.** Two connections run `CREATE TABLE IF NOT EXISTS` for the same table at once, 20 times. Record any error\n text, and the catch rule that makes it idempotent.\n - **Q8.** Bucket statistics: one `GROUP BY` over the expression `(cast(ts AS LONG)*10 - $1) / $2`, with\n `sum(CASE WHEN \u2026 THEN 1 ELSE 0 END)`, `avg`, `min`, `max`, `sum`. Plus the first and last usable sample per bucket,\n with its value and time. Try `first()`/`last()` with a filter, or a window `row_number() OVER (PARTITION BY b\n ORDER BY ts)`. Plus arg-min/arg-max time, via a window function or a second statement.\n - **Q9.** AtTime neighbour lookup: chunked `LIMIT 1` seeks (`ts <= t ORDER BY ts DESC LIMIT 1` and\n `ts > t ORDER BY ts LIMIT 1`, combined with `UNION ALL` per time) versus `ASOF JOIN`. Time both for 100 timestamps\n and pick one.\n - **Q10.** VARCHAR and SYMBOL accept a 400-character string, and `SYMBOL` capacity is fine for tens of thousands of\n tags.\n - **Q11.** An existing-timestamp lookup for a batch: `WHERE tag=$1 AND ts >= $2 AND ts <= $3` (a range), or an `IN`\n list. Pick the one that works with parameters.\n - **Q12.** `DROP TABLE IF EXISTS` works for test cleanup, and `tables()` lists the tables with a given prefix.\n\n- [ ] **Step 3: Write the spike doc** with the image tag and date, the connection string, and Q1\u2013Q12 (SQL, output and\n decision for each).\n- [ ] **Step 4: Amend the spec \u00a72\u2013\u00a74** in place with the confirmed forms. If any assumption failed, write the\n replacement design inline and add a ledger ruling. If the failure breaks a user decision, stop and ask.\n- [ ] **Step 5: Commit**\n\n```bash\ngit add docker-dev/docker-compose.questdb.yml docs/plans/2026-10-10-questdb-spike.md docs/plans/2026-10-10-questdb-historian-design.md\ngit commit -m \"spike(historian): QuestDB behaviours confirmed against pinned image; spec \u00a72\u2013\u00a74 amended\n\nClaude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe\"\n```\n\n```json:metadata\n{\"files\":[\"docker-dev/docker-compose.questdb.yml\",\"docs/plans/2026-10-10-questdb-spike.md\",\"docs/plans/2026-10-10-questdb-historian-design.md\"],\"verifyCommand\":\"docker exec otopcua-dev-questdb-1 true\",\"acceptanceCriteria\":[\"pinned image \u22652 weeks old\",\"overlay ports 18812/19000\",\"Q1\u2013Q12 answered with output\",\"spec \u00a72\u2013\u00a74 amended\"],\"modelTier\":\"frontier\",\"model\":\"controller\",\"wave\":0}\n```\n\n---" }, { "id": 1, "subject": "Task 1: Extract `Driver.Historian.Sql.Common` (no behaviour cha", "status": "pending", - "description": "**Goal:** move every database-independent piece of `Driver.Historian.SqlServer` into a new shared library. The SQL\nServer backend then calls it, with byte-identical behaviour.\n\n**Files:**\n- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.csproj`\n- Create in that project: `HistorianStatusCodes.cs`, `StatusBits.cs`, `HistorianInfoBits.cs`, `HistorianDataTypes.cs`,\n `SampleColumns.cs`, `HistorianSampleRow.cs`, `StoredSample.cs`, `BucketRow.cs`, `AggregateFamily.cs`,\n `AggregateEvaluator.cs`, `AtTimeEvaluator.cs`, `BucketPlan.cs`, `LiveWritePreparer.cs`, `HistoryUpdatePlanner.cs`,\n `HistorianTagNames.cs`, `BootstrapFailureCache.cs`, `HistorianSchemaMissingException.cs`, `AlarmEventIds.cs`\n- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/` (csproj + tests)\n- Modify: every file in `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/` that contained the moved code\n (`SqlServerHistorianDataSource.cs`, `SqlAggregateQueries.cs`, `SqlHistorianDataTypes.cs` (delete),\n `SqlHistorianStatusCodes.cs` (keep only SQL Server-only codes, or delete if none), `SampleRowMapper.cs`,\n `SqlServerHistorianValueWriter.cs`, `SqlServerHistorianDataUpdater.cs`, `SqlServerTagCatalog.cs`,\n `SqlServerSchemaBootstrapper.cs`, the csproj)\n- Modify: the SqlServer unit tests that tested moved code. Move them into `Sql.Common.Tests` under the same test name.\n- Modify: `ZB.MOM.WW.OtOpcUa.slnx` (add both new projects alphabetically, next to lines 30 and 104)\n- Modify: `tests/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions.Tests/ZB.MOM.WW.OtOpcUa.Core.Abstractions.Tests.csproj` (add a\n ProjectReference to Sql.Common)\n\n**Acceptance Criteria:**\n- [ ] `Sql.Common` has no reference to `Microsoft.Data.SqlClient` or any other database client. Its references are\n Core.Abstractions, Server/Runtime (only if needed for `HistorizationValue`) and\n `Microsoft.Extensions.Logging.Abstractions`.\n- [ ] The public API matches the signatures in Step 1 exactly. T2\u2013T7 depend on them.\n- [ ] The SqlServer integration suite passes with the same test count as before the change, which is recorded in the\n report. The SqlServer unit suite plus the moved tests in `Sql.Common.Tests` has the same total count.\n- [ ] New `Sql.Common.Tests` unit tests cover `AtTimeEvaluator`, `AggregateEvaluator`, `BucketPlan`, `LiveWritePreparer`\n and `HistoryUpdatePlanner` directly, using the literals from the SqlServer integration tests. Each one is\n falsified once.\n- [ ] StatusCodeParity passes, and its constant count does not drop.\n\n**Verify:**\n- `dotnet build ZB.MOM.WW.OtOpcUa.slnx` gives 0 errors.\n- `dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests` passes.\n- `dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.Tests` passes.\n- With `HISTORIAN_SQL_TEST_CONNECTION` set, `dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests`\n passes with the same count as before.\n- `dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions.Tests --filter \"FullyQualifiedName~StatusCodeParity\"`\n passes.\n\n**Steps:**\n\n- [ ] **Step 0: Record the baseline.** Run both SqlServer test projects and the parity filter, and write the exact\n pass counts into the report.\n\n- [ ] **Step 1: Create the shared API.** Namespace: `ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common`. Each moved body is\n transplanted verbatim from the source named in the comment; only names and visibility change.\n\n```csharp\n// HistorianStatusCodes.cs \u2014 from SqlHistorianStatusCodes + SqlHistorianUpdateStatusCodes (all consts, same values)\npublic static class HistorianStatusCodes\n{\n public const uint Good = 0x00000000u, GoodEntryInserted = 0x00A20000u, GoodEntryReplaced = 0x00A30000u;\n public const uint BadNoData = 0x809B0000u, BadEntryExists = 0x809F0000u, BadNoEntryExists = 0x80A00000u;\n public const uint BadInvalidTimestamp = 0x80230000u, BadAggregateInvalidInputs = 0x80D60000u;\n public const uint UncertainDataSubNormal = 0x40A40000u, BadOutOfRange = 0x803C0000u, BadInvalidArgument = 0x80AB0000u;\n}\n// StatusBits.cs / HistorianInfoBits.cs \u2014 moved verbatim from SqlHistorianStatusCodes.cs (made public)\n// HistorianDataTypes.cs \u2014 from SqlHistorianDataTypes: DefaultForEmpty, byte FromKind(HistorianValueKind)\n// SampleColumns.cs \u2014 record struct SampleColumns(byte ValueKind, double? NumValue, long? IntValue, string? StrValue)\n// + static class SampleColumnMapper { ToColumns(HistorianValue); FromColumns(byte,double?,long?,string?);\n// int ToDbStatus(uint); uint FromDbStatus(int); long ToLongStatus(uint) => status; uint FromLongStatus(long) }\n// HistorianSampleRow.cs \u2014 record struct HistorianSampleRow(DateTime TimeUtc, HistorianValue Value, uint StatusCode) (was SqlSampleRow)\npublic readonly record struct StoredSample(DateTime TimeUtc, HistorianValue Value, uint Status)\n{\n public bool IsUsable { get; } // verbatim rule from SqlServerHistorianDataSource\n public bool IsGood { get; }\n /// Applies the quality clamp (NULL value + non-Bad \u21d2 BadNoData) \u2014 from ReadStored line 533.\n public static StoredSample FromStored(DateTime timeUtc, HistorianValue value, uint status);\n}\npublic sealed class BucketRow // fields verbatim from the private BucketRow, as public get/set properties\n{\n public long Bucket { get; set; } public int Total { get; set; } public int Usable { get; set; }\n public int Good { get; set; } public int Bad { get; set; } public int NonNumeric { get; set; }\n public double? Avg { get; set; } public double? Min { get; set; } public double? Max { get; set; } public double? Sum { get; set; }\n public HistorianValue First { get; set; } public HistorianValue Last { get; set; }\n public double? Extreme { get; set; } public DateTime? ExtremeTime { get; set; }\n}\npublic enum AggregateFamily { Simple, StartEnd, ActualTime }\npublic static class AggregateFamilies { public static AggregateFamily? FamilyOf(HistoryAggregateType a); } // from SqlAggregateQueries.FamilyOf\npublic static class AggregateEvaluator\n{\n public static DataValueSnapshot Evaluate(HistoryAggregateType a, BucketRow r, DateTime bucketStart, DateTime now); // verbatim\n public static DataValueSnapshot NoData(DateTime t, DateTime now); // verbatim\n}\npublic static class AtTimeEvaluator\n{\n public static DataValueSnapshot Evaluate(DateTime t, StoredSample? prev, StoredSample? next, DateTime serverTime, DateTime now); // verbatim AtTimeValue\n}\npublic sealed class BucketPlan // from ReadProcessedAsync lines 355-370 and 395-408\n{\n public const int MaxBuckets = 100_000;\n public DateTime StartUtc { get; } public DateTime EndUtc { get; } public bool Reversed { get; }\n public long IntervalTicks { get; } public long BucketCount { get; }\n public long StartUnixTicks => (StartUtc - DateTime.UnixEpoch).Ticks; // for QuestDB's (cast(ts AS LONG)*10 - start)/interval\n /// BucketCount > MaxBuckets (param name \"interval\") \u2014 same as today.\n public static BucketPlan Create(DateTime startUtc, DateTime endUtc, TimeSpan interval);\n public DateTime BucketStart(long index);\n /// One snapshot per bucket in window order; missing buckets get ; reversed if the window was.\n public IReadOnlyList Assemble(IReadOnlyDictionary byBucket, Func empty);\n}\npublic static class LiveWritePreparer // from SqlServerHistorianValueWriter.Prepare, plus a resolution parameter\n{\n /// 1 = keep 100 ns (SQL Server); 10 = truncate to \u00b5s BEFORE de-duplication (QuestDB).\n public static List Prepare(IReadOnlyList values, Func utcNow, long resolutionTicks = 1);\n}\npublic enum UpsertOutcome { Inserted, Replaced }\npublic sealed class UpdatePlan\n{\n public uint?[] Results { get; } // pre-decided codes (invalid ts, NaN, superseded duplicate\u2026); null = decided by the backend\n public IReadOnlyList<(int Index, HistorianSampleRow Row)> Winners { get; } // rows to send, de-duplicated per mode\n}\npublic static class HistoryUpdatePlanner // pre/post passes from SqlServerHistorianDataUpdater.UpdateDataAsync lines 56-108\n{\n public static UpdatePlan Plan(string tag, HistoryUpdateMode mode, IReadOnlyList values, long resolutionTicks = 1);\n /// ticks of each winner that the backend actually wrote \u2192 Inserted/Replaced; absent \u21d2 not written.\n public static IReadOnlyList Complete(UpdatePlan plan, HistoryUpdateMode mode, IReadOnlyDictionary outcomeByTick);\n public static IReadOnlyList DeleteAtTimeResults(IReadOnlyList times, ISet deletedTicks, bool tagKnown);\n public static uint DeleteRawResult(bool tagKnown, long rowsAffected);\n}\npublic static class HistorianTagNames { public const int MaxLength = 400; public static bool IsValid(string? name); } // from SqlServerTagCatalog.IsValidName\npublic sealed class BootstrapFailureCache // the 5 s negative-cache mechanism from SqlServerSchemaBootstrapper\n{\n public BootstrapFailureCache(TimeProvider clock, TimeSpan duration);\n public void ThrowIfCached(); public void Record(Exception ex); public void Clear();\n}\n// HistorianSchemaMissingException.cs \u2014 moved from SqlServerHistorianOptions.cs (public sealed, ctor(string message))\n// AlarmEventIds.cs \u2014 public static string ComputeEventId(...) moved verbatim from SqlServerAlarmHistorianWriter (same signature)\n```\n\n- [ ] **Step 2: Point the SqlServer backend at the shared lib.**\n - Replace each moved body with a call into `Sql.Common`.\n - Delete the now-empty SqlServer classes.\n - Keep `SqlAggregateQueries`' SQL text and `ReadBucket`, now filling the shared `BucketRow`.\n - Keep `SampleRowMapper`'s TVP builders, which take `HistorianSampleRow`.\n- [ ] **Step 3: Move the SqlServer unit tests for the moved code** into `Sql.Common.Tests`, keeping test names and\n literals: `StatusCodeOracleTests`, `ValueWriterPrepareTests`, the column round-trip parts of\n `SampleRowMapperTests`, and the data-type table test. The TVP parts stay in SqlServer.Tests.\n- [ ] **Step 4: Add direct unit tests that did not exist before.** For example:\n\n```csharp\n[Fact] public void AtTime_inside_an_outage_returns_the_bad_neighbours_code_with_no_value()\n{\n var t0 = new DateTime(2026,10,9,8,0,0,DateTimeKind.Utc);\n var prev = new StoredSample(t0, HistorianValue.Empty, 0x80310000u);\n var next = new StoredSample(t0.AddSeconds(10), HistorianValue.FromDouble(5), 0);\n var r = AtTimeEvaluator.Evaluate(t0.AddSeconds(5), prev, next, t0, t0.AddHours(1));\n r.StatusCode.ShouldBe(0x80310000u); r.Value.ShouldBeNull();\n}\n[Fact] public void BucketPlan_rejects_more_than_100k_buckets()\n => Should.Throw(() =>\n BucketPlan.Create(DateTime.UnixEpoch, DateTime.UnixEpoch.AddSeconds(100_001), TimeSpan.FromSeconds(1)));\n[Fact] public void Prepare_truncates_to_microseconds_before_dedup_when_resolution_is_10()\n{\n var t = new DateTime(2026,10,9,8,0,0,DateTimeKind.Utc).AddTicks(1234567);\n var rows = LiveWritePreparer.Prepare([new(t, HistorianValue.FromDouble(1), 0), new(t.AddTicks(3), HistorianValue.FromDouble(2), 0)], () => t, 10);\n rows.Count.ShouldBe(1); rows[0].TimeUtc.Ticks.ShouldBe(t.Ticks - t.Ticks % 10); rows[0].Value.Number.ShouldBe(1);\n}\n```\n\n Add the matching `AggregateEvaluator` cases, using the 8-row ProcessedRead literals from\n `ProcessedReadTests.Numeric_aggregates_over_the_fixture`. Build the `BucketRow`s by hand from tag A's samples.\n- [ ] **Step 5: Wire the solution and parity test.** Add both projects to the slnx, and the ProjectReference to the\n Core.Abstractions.Tests csproj.\n- [ ] **Step 6: Run the Verify commands.** Counts must equal the Step 0 baseline, plus the new direct tests. Falsify\n each new direct test once by reverting the rule it pins.\n- [ ] **Step 7: Commit** with `refactor(historian): extract Driver.Historian.Sql.Common from the SQL Server backend (no\n behaviour change)` and the Claude-Session trailer.\n\n```json:metadata\n{\"files\":[\"src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/\",\"tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/\",\"src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/\",\"tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.Tests/\",\"ZB.MOM.WW.OtOpcUa.slnx\",\"tests/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions.Tests/ZB.MOM.WW.OtOpcUa.Core.Abstractions.Tests.csproj\"],\"verifyCommand\":\"dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests (env set) \u2014 same count as baseline\",\"acceptanceCriteria\":[\"no DB client ref in Sql.Common\",\"public API exactly as Step 1\",\"SqlServer suites same counts\",\"direct evaluator tests falsified\",\"parity green\"],\"modelTier\":\"frontier\",\"model\":\"opus\",\"wave\":1}\n```\n\n---", + "description": "> **Review amendments (Fable) \u2014 binding:**\n> - Runs as dispatch **1a** then **1b** (R10).\n> - **Files gain** `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/**`, for mechanical\n> rename and using fixes only (R11 / B1). This includes replacing `SqlServerSchemaBootstrapper.FailureCacheDuration`\n> with the `BootstrapFailureCache` constant.\n> - `ProcessedReadTests.Every_enum_member_is_either_served_or_refused` moves to `Sql.Common.Tests` (B2 list).\n> - **Step 1 API additions (S6):**\n> - `AlarmEventIds.ToNumericSeverity(AlarmSeverity)` (moved from the private SqlServer method);\n> - `public sealed class HistorianHealthCounters` (lock-guarded): `RecordSuccess()`, `RecordFailure(Exception)`\n> (`OperationCanceledException` is not a failure), and `Snapshot()` returning `HistorianHealthSnapshot` with the\n> `ProcessConnectionOpen = successes > 0 && consecutive == 0` rule. `SqlServerHistorianDataSource` uses it too.\n> - `public static class EventContinuation { string Encode(DateTime, string eventId) }`, in the existing\n> `\"{ticks}|{EventId}\"` format.\n> - `HistoricalEventRows.ClampSeverity(int) \u2192 ushort`.\n> - `StatusBits.SeverityUncertainBit = 0x40000000u`.\n> - Copy `Platforms AnyCPU;x64` into the new csproj (N4).\n\n\n**Goal:** move every database-independent piece of `Driver.Historian.SqlServer` into a new shared library. The SQL\nServer backend then calls it, with byte-identical behaviour.\n\n**Files:**\n- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.csproj`\n- Create in that project: `HistorianStatusCodes.cs`, `StatusBits.cs`, `HistorianInfoBits.cs`, `HistorianDataTypes.cs`,\n `SampleColumns.cs`, `HistorianSampleRow.cs`, `StoredSample.cs`, `BucketRow.cs`, `AggregateFamily.cs`,\n `AggregateEvaluator.cs`, `AtTimeEvaluator.cs`, `BucketPlan.cs`, `LiveWritePreparer.cs`, `HistoryUpdatePlanner.cs`,\n `HistorianTagNames.cs`, `BootstrapFailureCache.cs`, `HistorianSchemaMissingException.cs`, `AlarmEventIds.cs`\n- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/` (csproj + tests)\n- Modify: every file in `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/` that contained the moved code\n (`SqlServerHistorianDataSource.cs`, `SqlAggregateQueries.cs`, `SqlHistorianDataTypes.cs` (delete),\n `SqlHistorianStatusCodes.cs` (keep only SQL Server-only codes, or delete if none), `SampleRowMapper.cs`,\n `SqlServerHistorianValueWriter.cs`, `SqlServerHistorianDataUpdater.cs`, `SqlServerTagCatalog.cs`,\n `SqlServerSchemaBootstrapper.cs`, the csproj)\n- Modify: the SqlServer unit tests that tested moved code. Move them into `Sql.Common.Tests` under the same test name.\n- Modify: `ZB.MOM.WW.OtOpcUa.slnx` (add both new projects alphabetically, next to lines 30 and 104)\n- Modify: `tests/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions.Tests/ZB.MOM.WW.OtOpcUa.Core.Abstractions.Tests.csproj` (add a\n ProjectReference to Sql.Common)\n\n**Acceptance Criteria:**\n- [ ] `Sql.Common` has no reference to `Microsoft.Data.SqlClient` or any other database client. Its references are\n Core.Abstractions, Server/Runtime (only if needed for `HistorizationValue`) and\n `Microsoft.Extensions.Logging.Abstractions`.\n- [ ] The public API matches the signatures in Step 1 exactly. T2\u2013T7 depend on them.\n- [ ] The SqlServer integration suite passes with the same test count as before the change, which is recorded in the\n report. The SqlServer unit suite plus the moved tests in `Sql.Common.Tests` has the same total count.\n- [ ] New `Sql.Common.Tests` unit tests cover `AtTimeEvaluator`, `AggregateEvaluator`, `BucketPlan`, `LiveWritePreparer`\n and `HistoryUpdatePlanner` directly, using the literals from the SqlServer integration tests. Each one is\n falsified once.\n- [ ] StatusCodeParity passes, and its constant count does not drop.\n\n**Verify:**\n- `dotnet build ZB.MOM.WW.OtOpcUa.slnx` gives 0 errors.\n- `dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests` passes.\n- `dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.Tests` passes.\n- With `HISTORIAN_SQL_TEST_CONNECTION` set, `dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests`\n passes with the same count as before.\n- `dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions.Tests --filter \"FullyQualifiedName~StatusCodeParity\"`\n passes.\n\n**Steps:**\n\n- [ ] **Step 0: Record the baseline.** Run both SqlServer test projects and the parity filter, and write the exact\n pass counts into the report.\n\n- [ ] **Step 1: Create the shared API.** Namespace: `ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common`. Each moved body is\n transplanted verbatim from the source named in the comment; only names and visibility change.\n\n```csharp\n// HistorianStatusCodes.cs \u2014 from SqlHistorianStatusCodes + SqlHistorianUpdateStatusCodes (all consts, same values)\npublic static class HistorianStatusCodes\n{\n public const uint Good = 0x00000000u, GoodEntryInserted = 0x00A20000u, GoodEntryReplaced = 0x00A30000u;\n public const uint BadNoData = 0x809B0000u, BadEntryExists = 0x809F0000u, BadNoEntryExists = 0x80A00000u;\n public const uint BadInvalidTimestamp = 0x80230000u, BadAggregateInvalidInputs = 0x80D60000u;\n public const uint UncertainDataSubNormal = 0x40A40000u, BadOutOfRange = 0x803C0000u, BadInvalidArgument = 0x80AB0000u;\n}\n// StatusBits.cs / HistorianInfoBits.cs \u2014 moved verbatim from SqlHistorianStatusCodes.cs (made public)\n// HistorianDataTypes.cs \u2014 from SqlHistorianDataTypes: DefaultForEmpty, byte FromKind(HistorianValueKind)\n// SampleColumns.cs \u2014 record struct SampleColumns(byte ValueKind, double? NumValue, long? IntValue, string? StrValue)\n// + static class SampleColumnMapper { ToColumns(HistorianValue); FromColumns(byte,double?,long?,string?);\n// int ToDbStatus(uint); uint FromDbStatus(int); long ToLongStatus(uint) => status; uint FromLongStatus(long) }\n// HistorianSampleRow.cs \u2014 record struct HistorianSampleRow(DateTime TimeUtc, HistorianValue Value, uint StatusCode) (was SqlSampleRow)\npublic readonly record struct StoredSample(DateTime TimeUtc, HistorianValue Value, uint Status)\n{\n public bool IsUsable { get; } // verbatim rule from SqlServerHistorianDataSource\n public bool IsGood { get; }\n /// Applies the quality clamp (NULL value + non-Bad \u21d2 BadNoData) \u2014 from ReadStored line 533.\n public static StoredSample FromStored(DateTime timeUtc, HistorianValue value, uint status);\n}\npublic sealed class BucketRow // fields verbatim from the private BucketRow, as public get/set properties\n{\n public long Bucket { get; set; } public int Total { get; set; } public int Usable { get; set; }\n public int Good { get; set; } public int Bad { get; set; } public int NonNumeric { get; set; }\n public double? Avg { get; set; } public double? Min { get; set; } public double? Max { get; set; } public double? Sum { get; set; }\n public HistorianValue First { get; set; } public HistorianValue Last { get; set; }\n public double? Extreme { get; set; } public DateTime? ExtremeTime { get; set; }\n}\npublic enum AggregateFamily { Simple, StartEnd, ActualTime }\npublic static class AggregateFamilies { public static AggregateFamily? FamilyOf(HistoryAggregateType a); } // from SqlAggregateQueries.FamilyOf\npublic static class AggregateEvaluator\n{\n public static DataValueSnapshot Evaluate(HistoryAggregateType a, BucketRow r, DateTime bucketStart, DateTime now); // verbatim\n public static DataValueSnapshot NoData(DateTime t, DateTime now); // verbatim\n}\npublic static class AtTimeEvaluator\n{\n public static DataValueSnapshot Evaluate(DateTime t, StoredSample? prev, StoredSample? next, DateTime serverTime, DateTime now); // verbatim AtTimeValue\n}\npublic sealed class BucketPlan // from ReadProcessedAsync lines 355-370 and 395-408\n{\n public const int MaxBuckets = 100_000;\n public DateTime StartUtc { get; } public DateTime EndUtc { get; } public bool Reversed { get; }\n public long IntervalTicks { get; } public long BucketCount { get; }\n public long StartUnixTicks => (StartUtc - DateTime.UnixEpoch).Ticks; // for QuestDB's (cast(ts AS LONG)*10 - start)/interval\n /// BucketCount > MaxBuckets (param name \"interval\") \u2014 same as today.\n public static BucketPlan Create(DateTime startUtc, DateTime endUtc, TimeSpan interval);\n public DateTime BucketStart(long index);\n /// One snapshot per bucket in window order; missing buckets get ; reversed if the window was.\n public IReadOnlyList Assemble(IReadOnlyDictionary byBucket, Func empty);\n}\npublic static class LiveWritePreparer // from SqlServerHistorianValueWriter.Prepare, plus a resolution parameter\n{\n /// 1 = keep 100 ns (SQL Server); 10 = truncate to \u00b5s BEFORE de-duplication (QuestDB).\n public static List Prepare(IReadOnlyList values, Func utcNow, long resolutionTicks = 1);\n}\npublic enum UpsertOutcome { Inserted, Replaced }\npublic sealed class UpdatePlan\n{\n public uint?[] Results { get; } // pre-decided codes (invalid ts, NaN, superseded duplicate\u2026); null = decided by the backend\n public IReadOnlyList<(int Index, HistorianSampleRow Row)> Winners { get; } // rows to send, de-duplicated per mode\n}\npublic static class HistoryUpdatePlanner // pre/post passes from SqlServerHistorianDataUpdater.UpdateDataAsync lines 56-108\n{\n public static UpdatePlan Plan(string tag, HistoryUpdateMode mode, IReadOnlyList values, long resolutionTicks = 1);\n /// ticks of each winner that the backend actually wrote \u2192 Inserted/Replaced; absent \u21d2 not written.\n public static IReadOnlyList Complete(UpdatePlan plan, HistoryUpdateMode mode, IReadOnlyDictionary outcomeByTick);\n public static IReadOnlyList DeleteAtTimeResults(IReadOnlyList times, ISet deletedTicks, bool tagKnown);\n public static uint DeleteRawResult(bool tagKnown, long rowsAffected);\n}\npublic static class HistorianTagNames { public const int MaxLength = 400; public static bool IsValid(string? name); } // from SqlServerTagCatalog.IsValidName\npublic sealed class BootstrapFailureCache // the 5 s negative-cache mechanism from SqlServerSchemaBootstrapper\n{\n public BootstrapFailureCache(TimeProvider clock, TimeSpan duration);\n public void ThrowIfCached(); public void Record(Exception ex); public void Clear();\n}\n// HistorianSchemaMissingException.cs \u2014 moved from SqlServerHistorianOptions.cs (public sealed, ctor(string message))\n// AlarmEventIds.cs \u2014 public static string ComputeEventId(...) moved verbatim from SqlServerAlarmHistorianWriter (same signature)\n```\n\n- [ ] **Step 2: Point the SqlServer backend at the shared lib.**\n - Replace each moved body with a call into `Sql.Common`.\n - Delete the now-empty SqlServer classes.\n - Keep `SqlAggregateQueries`' SQL text and `ReadBucket`, now filling the shared `BucketRow`.\n - Keep `SampleRowMapper`'s TVP builders, which take `HistorianSampleRow`.\n- [ ] **Step 3: Move the SqlServer unit tests for the moved code** into `Sql.Common.Tests`, keeping test names and\n literals: `StatusCodeOracleTests`, `ValueWriterPrepareTests`, the column round-trip parts of\n `SampleRowMapperTests`, and the data-type table test. The TVP parts stay in SqlServer.Tests.\n- [ ] **Step 4: Add direct unit tests that did not exist before.** For example:\n\n```csharp\n[Fact] public void AtTime_inside_an_outage_returns_the_bad_neighbours_code_with_no_value()\n{\n var t0 = new DateTime(2026,10,9,8,0,0,DateTimeKind.Utc);\n var prev = new StoredSample(t0, HistorianValue.Empty, 0x80310000u);\n var next = new StoredSample(t0.AddSeconds(10), HistorianValue.FromDouble(5), 0);\n var r = AtTimeEvaluator.Evaluate(t0.AddSeconds(5), prev, next, t0, t0.AddHours(1));\n r.StatusCode.ShouldBe(0x80310000u); r.Value.ShouldBeNull();\n}\n[Fact] public void BucketPlan_rejects_more_than_100k_buckets()\n => Should.Throw(() =>\n BucketPlan.Create(DateTime.UnixEpoch, DateTime.UnixEpoch.AddSeconds(100_001), TimeSpan.FromSeconds(1)));\n[Fact] public void Prepare_truncates_to_microseconds_before_dedup_when_resolution_is_10()\n{\n var t = new DateTime(2026,10,9,8,0,0,DateTimeKind.Utc).AddTicks(1234567);\n var rows = LiveWritePreparer.Prepare([new(t, HistorianValue.FromDouble(1), 0), new(t.AddTicks(3), HistorianValue.FromDouble(2), 0)], () => t, 10);\n rows.Count.ShouldBe(1); rows[0].TimeUtc.Ticks.ShouldBe(t.Ticks - t.Ticks % 10); rows[0].Value.Number.ShouldBe(1);\n}\n```\n\n Add the matching `AggregateEvaluator` cases, using the 8-row ProcessedRead literals from\n `ProcessedReadTests.Numeric_aggregates_over_the_fixture`. Build the `BucketRow`s by hand from tag A's samples.\n- [ ] **Step 5: Wire the solution and parity test.** Add both projects to the slnx, and the ProjectReference to the\n Core.Abstractions.Tests csproj.\n- [ ] **Step 6: Run the Verify commands.** Counts must equal the Step 0 baseline, plus the new direct tests. Falsify\n each new direct test once by reverting the rule it pins.\n- [ ] **Step 7: Commit** with `refactor(historian): extract Driver.Historian.Sql.Common from the SQL Server backend (no\n behaviour change)` and the Claude-Session trailer.\n\n```json:metadata\n{\"files\":[\"src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/\",\"tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/\",\"src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/\",\"tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.Tests/\",\"ZB.MOM.WW.OtOpcUa.slnx\",\"tests/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions.Tests/ZB.MOM.WW.OtOpcUa.Core.Abstractions.Tests.csproj\"],\"verifyCommand\":\"dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests (env set) \u2014 same count as baseline\",\"acceptanceCriteria\":[\"no DB client ref in Sql.Common\",\"public API exactly as Step 1\",\"SqlServer suites same counts\",\"direct evaluator tests falsified\",\"parity green\"],\"modelTier\":\"frontier\",\"model\":\"opus\",\"wave\":1}\n```\n\n---", "blockedBy": [ 0 ] @@ -20,7 +20,7 @@ "id": 2, "subject": "Task 2: Conformance suite (shared source), run against SQL Serv", "status": "pending", - "description": "**Goal:** one backend-neutral suite of hand-computed behaviour tests, compiled into the SqlServer integration project\nand passing there. T7 later compiles the same suite into the QuestDB project.\n\n**Files:**\n- Create: `tests/Drivers/Historian.Conformance/IConformanceBackend.cs`, `ConformanceData.cs`,\n `RawReadConformance.cs`, `AtTimeConformance.cs`, `ProcessedConformance.cs`, `QualityContractConformance.cs`,\n `LiveWriteConformance.cs`, `UpdateConformance.cs`, `TagNameConformance.cs`\n- Modify: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/*.csproj` (add\n `` and a ProjectReference to Sql.Common)\n- Create in the SqlServer IT project: `SqlServerConformanceBackend.cs`, `SqlServerConformance.cs` (one concrete\n subclass per conformance class)\n- Modify in the SqlServer IT project: remove from the original files the generic tests that moved into the suite\n (`RawReadTests`, `AtTimeReadTests`, `ProcessedReadTests`, `QualityContractTests`, and the generic parts of\n `WriteTests`, `UpdateTests` and `LongNameTests`). SQL Server-specific tests stay where they are.\n\n**Acceptance Criteria:**\n- [ ] Every moved test keeps its method name and literal expectations. `git log -p` shows a move, not a rewrite.\n- [ ] The SqlServer IT total count after the change is at least the pre-change count, and the report has the before\n and after count.\n- [ ] Suite code references only `IConformanceBackend` and Core.Abstractions types. There is no `SqlServer*` type and\n no T-SQL in `tests/Drivers/Historian.Conformance`.\n- [ ] Timestamp assertions go through `B.Round(t)`.\n- [ ] Any test that cannot hold for every backend is a `virtual` member with a doc comment naming the difference.\n None should be needed for SQL Server.\n\n**Verify:** with `HISTORIAN_SQL_TEST_CONNECTION` set, `dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests`\npasses with a count of at least the baseline. Without the variable, every test skips.\n\n**Steps:**\n\n- [ ] **Step 1: Define the backend contract.**\n\n```csharp\nnamespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Conformance;\npublic interface IConformanceBackend : IAsyncDisposable\n{\n /// 1 for SQL Server (100 ns), 10 for QuestDB (\u00b5s).\n long ResolutionTicks { get; }\n DateTime Round(DateTime t) => new(t.Ticks - t.Ticks % ResolutionTicks, DateTimeKind.Utc);\n IHistorianDataSource CreateDataSource(TimeProvider? clock = null);\n IHistorianValueWriter CreateValueWriter(Func? utcNow = null);\n IHistorianDataUpdater CreateUpdater();\n /// Writes rows directly in the backend's schema (never through the writer), status verbatim.\n Task SeedAsync(string tag, params (DateTime T, HistorianValue V, uint Status)[] samples);\n /// Live (non-deleted) rows for a tag, ascending, read directly from storage.\n Task> ReadBackAsync(string tag);\n}\npublic abstract class ConformanceBase : IAsyncLifetime\n{\n protected IConformanceBackend B { get; private set; } = null!;\n protected abstract Task CreateBackendAsync(); // calls fixture.SkipIfUnavailable()\n public async ValueTask InitializeAsync() => B = await CreateBackendAsync();\n public async ValueTask DisposeAsync() => await B.DisposeAsync();\n}\n```\n\n `ConformanceData` holds `T0`, `At(seconds)`, the status consts and `SeedAAsync(B)`, verbatim from `ReadTestDb`.\n- [ ] **Step 2: Move the generic tests** listed in the T1 inventory (Raw, AtTime, Processed literals, the quality\n contract, live-write redelivery and first-wins, update result codes, tag names over 400 characters) into the\n suite classes. Write each one as an abstract class `RawReadConformance : ConformanceBase` with `[Fact]` methods.\n- [ ] **Step 3: Implement `SqlServerConformanceBackend`.** It reuses `SqlHistorianFixture`, a unique schema and the\n existing seeding SQL from `ReadTestDb`. Add one concrete class per suite:\n `[Collection(SqlHistorianCollection.Name)] public sealed class SqlServerRawReadConformance(SqlHistorianFixture fx) : RawReadConformance`.\n- [ ] **Step 4: Run the tests and compare counts.** Commit with\n `test(historian): backend conformance suite (shared source), run against SQL Server` and the trailer.\n\n```json:metadata\n{\"files\":[\"tests/Drivers/Historian.Conformance/\",\"tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/\"],\"verifyCommand\":\"dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests (env set) \u2265 baseline count\",\"acceptanceCriteria\":[\"tests moved by name with same literals\",\"count \u2265 baseline\",\"suite is backend-neutral\",\"Round() for timestamps\"],\"modelTier\":\"standard\",\"model\":\"sonnet\",\"wave\":2}\n```\n\n---", + "description": "> **Review amendments (Fable) \u2014 binding:**\n> - **AC bullet 1 becomes:** \"Every moved test keeps its method name and its expected VALUES/STATUS literals. Tick\n> offsets are rewritten per R4. The report lists each test whose offsets were rewritten.\"\n> - **AC bullet 5 becomes:** \"A `virtual` override is allowed only for a difference R4 cannot express. The report names\n> it, and the controller rules on it before T7.\"\n> - **Step 2 explicit list (B2). Move:**\n> - `RawReadTests` \u2014 all except `A_missing_schema_surfaces_in_health_LastError_and_is_not_latched`. The SQL tag-count\n> line in `Unknown_tag_is_an_empty_result_and_reads_never_create_tags` becomes `B.TagExistsAsync`.\n> - `AtTimeReadTests` \u2014 all.\n> - `ProcessedReadTests` \u2014 all except `Every_enum_member_is_either_served_or_refused`, which went to Sql.Common.Tests\n> in T1.\n> - `QualityContractTests` \u2014 all. Direct status-column reads become `B.ReadBackAsync` status checks.\n> - `WriteTests` \u2014 `Every_kind_round_trips\u2026`, `Writing_the_same_batch_twice_is_idempotent`,\n> `Within_batch_duplicate_timestamps\u2026`, `Non_finite_doubles\u2026`, `Empty_batch_succeeds\u2026` and `Tag_is_created\u2026`.\n> - `UpdateTests` \u2014 all.\n> - `LongNameTests` \u2014 all except `An_event_read_filtered_by_a_long_source_matches_nothing`.\n> - **Stay in SqlServer:**\n> - `A_missing_schema\u2026`\n> - `Ten_thousand_values_write_in_under_five_seconds`\n> - `Concurrent_overlapping_writers\u2026`\n> - `A_SQL_failure_returns_false\u2026`\n> - `EventReadTests`\n> - `ProvisioningTests`\n> - `AlarmWriterTests`\n> - `SchemaBootstrapIntegrationTests`\n> - **Interface (S13):** add `Task TagExistsAsync(string tag);`. `ReadBackAsync` returns the stored status\n> verbatim, with no read-side clamp; say so in its doc comment.\n> - **Skip check (S7):** first run with the env var UNSET and confirm the tests are reported Skipped, not Failed.\n> Fallback: `ConformanceBase` exposes a lazy, memoised `protected async Task BackendAsync()`,\n> disposed in `DisposeAsync`, and each test starts with `var B = await BackendAsync();`.\n> - **Verify (N8):** the report lists every moved method name, plus per-file `[Fact]`/`[Theory]` counts before and\n> after.\n\n\n**Goal:** one backend-neutral suite of hand-computed behaviour tests, compiled into the SqlServer integration project\nand passing there. T7 later compiles the same suite into the QuestDB project.\n\n**Files:**\n- Create: `tests/Drivers/Historian.Conformance/IConformanceBackend.cs`, `ConformanceData.cs`,\n `RawReadConformance.cs`, `AtTimeConformance.cs`, `ProcessedConformance.cs`, `QualityContractConformance.cs`,\n `LiveWriteConformance.cs`, `UpdateConformance.cs`, `TagNameConformance.cs`\n- Modify: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/*.csproj` (add\n `` and a ProjectReference to Sql.Common)\n- Create in the SqlServer IT project: `SqlServerConformanceBackend.cs`, `SqlServerConformance.cs` (one concrete\n subclass per conformance class)\n- Modify in the SqlServer IT project: remove from the original files the generic tests that moved into the suite\n (`RawReadTests`, `AtTimeReadTests`, `ProcessedReadTests`, `QualityContractTests`, and the generic parts of\n `WriteTests`, `UpdateTests` and `LongNameTests`). SQL Server-specific tests stay where they are.\n\n**Acceptance Criteria:**\n- [ ] Every moved test keeps its method name and literal expectations. `git log -p` shows a move, not a rewrite.\n- [ ] The SqlServer IT total count after the change is at least the pre-change count, and the report has the before\n and after count.\n- [ ] Suite code references only `IConformanceBackend` and Core.Abstractions types. There is no `SqlServer*` type and\n no T-SQL in `tests/Drivers/Historian.Conformance`.\n- [ ] Timestamp assertions go through `B.Round(t)`.\n- [ ] Any test that cannot hold for every backend is a `virtual` member with a doc comment naming the difference.\n None should be needed for SQL Server.\n\n**Verify:** with `HISTORIAN_SQL_TEST_CONNECTION` set, `dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests`\npasses with a count of at least the baseline. Without the variable, every test skips.\n\n**Steps:**\n\n- [ ] **Step 1: Define the backend contract.**\n\n```csharp\nnamespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Conformance;\npublic interface IConformanceBackend : IAsyncDisposable\n{\n /// 1 for SQL Server (100 ns), 10 for QuestDB (\u00b5s).\n long ResolutionTicks { get; }\n DateTime Round(DateTime t) => new(t.Ticks - t.Ticks % ResolutionTicks, DateTimeKind.Utc);\n IHistorianDataSource CreateDataSource(TimeProvider? clock = null);\n IHistorianValueWriter CreateValueWriter(Func? utcNow = null);\n IHistorianDataUpdater CreateUpdater();\n /// Writes rows directly in the backend's schema (never through the writer), status verbatim.\n Task SeedAsync(string tag, params (DateTime T, HistorianValue V, uint Status)[] samples);\n /// Live (non-deleted) rows for a tag, ascending, read directly from storage.\n Task> ReadBackAsync(string tag);\n}\npublic abstract class ConformanceBase : IAsyncLifetime\n{\n protected IConformanceBackend B { get; private set; } = null!;\n protected abstract Task CreateBackendAsync(); // calls fixture.SkipIfUnavailable()\n public async ValueTask InitializeAsync() => B = await CreateBackendAsync();\n public async ValueTask DisposeAsync() => await B.DisposeAsync();\n}\n```\n\n `ConformanceData` holds `T0`, `At(seconds)`, the status consts and `SeedAAsync(B)`, verbatim from `ReadTestDb`.\n- [ ] **Step 2: Move the generic tests** listed in the T1 inventory (Raw, AtTime, Processed literals, the quality\n contract, live-write redelivery and first-wins, update result codes, tag names over 400 characters) into the\n suite classes. Write each one as an abstract class `RawReadConformance : ConformanceBase` with `[Fact]` methods.\n- [ ] **Step 3: Implement `SqlServerConformanceBackend`.** It reuses `SqlHistorianFixture`, a unique schema and the\n existing seeding SQL from `ReadTestDb`. Add one concrete class per suite:\n `[Collection(SqlHistorianCollection.Name)] public sealed class SqlServerRawReadConformance(SqlHistorianFixture fx) : RawReadConformance`.\n- [ ] **Step 4: Run the tests and compare counts.** Commit with\n `test(historian): backend conformance suite (shared source), run against SQL Server` and the trailer.\n\n```json:metadata\n{\"files\":[\"tests/Drivers/Historian.Conformance/\",\"tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/\"],\"verifyCommand\":\"dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests (env set) \u2265 baseline count\",\"acceptanceCriteria\":[\"tests moved by name with same literals\",\"count \u2265 baseline\",\"suite is backend-neutral\",\"Round() for timestamps\"],\"modelTier\":\"standard\",\"model\":\"sonnet\",\"wave\":2}\n```\n\n---", "blockedBy": [ 1 ] @@ -29,7 +29,7 @@ "id": 3, "subject": "Task 3: QuestDB skeleton (options, connection, bootstrapper, fi", "status": "pending", - "description": "**Goal:** a building `Driver.Historian.QuestDb` project. It has validated options, a pooled Npgsql connection, an\nidempotent and concurrency-safe schema bootstrapper with TTL, and an env-gated integration fixture.\n\n**Files:**\n- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.csproj`\n (references Core.Abstractions, Core.AlarmHistorian, Server/Runtime and Sql.Common; packages `Npgsql` and\n `Microsoft.Extensions.Logging.Abstractions`; InternalsVisibleTo the two QuestDb test assemblies)\n- Create: `QuestDbHistorianOptions.cs`, `QuestDbConnection.cs`, `QuestDbTables.cs`, `QuestDbSchemaBootstrapper.cs`\n- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests/` (csproj, `QuestDbHistorianOptionsTests.cs`,\n `QuestDbTablesTests.cs`)\n- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/` (csproj with the\n conformance-include line and a ProjectReference to Sql.Common, `QuestDbFixture.cs`, `BootstrapTests.cs`)\n- Modify: `ZB.MOM.WW.OtOpcUa.slnx`, and the Core.Abstractions.Tests csproj (ProjectReference to the QuestDb project)\n\n**Acceptance Criteria:**\n- [ ] `QuestDbHistorianOptions` has `ConnectionString` (default `\"\"`), `TablePrefix` (`\"hist_\"`, regex\n `^[A-Za-z0-9_]+$`), `RetentionDays` (`int?`, \u2265 1), `CommandTimeout` (30 s, > 0), `AutoCreateSchema` (`true`) and\n `VisibilityTimeout` (5 s, \u2265 0). `Validate()` returns error strings that name the key\n `ServerHistorian:QuestDb:`.\n- [ ] `QuestDbTables` produces the three `CREATE TABLE IF NOT EXISTS` statements, as confirmed by T0, for a given prefix\n and retention. Unit tests pin the exact text.\n- [ ] The bootstrapper is idempotent and survives two concurrent instances, using the catch rule from T0 Q7. It\n re-applies a changed TTL. It uses `BootstrapFailureCache` for the 5 s negative cache.\n- [ ] `QuestDbFixture` reads `HISTORIAN_QUESTDB_TEST_CONNECTION` and skips when it is absent. `UniquePrefix()` returns\n `\"t\" + 12 hex + \"_\"`. `DropPrefixAsync(prefix)` drops every table with that prefix, found through `tables()`.\n- [ ] `BootstrapTests` cover:\n - tables created;\n - a second run is a no-op;\n - two concurrent bootstrappers both succeed;\n - TTL is visible in metadata when `RetentionDays=7` and absent when unset;\n - `AutoCreateSchema=false` against a missing table throws `HistorianSchemaMissingException` from Sql.Common.\n\n**Verify:**\n- `dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests` passes.\n- With `HISTORIAN_QUESTDB_TEST_CONNECTION` set, `dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests --filter \"FullyQualifiedName~Bootstrap\"`\n passes.\n- Without the variable, every test skips.\n\n**Steps:**\n- [ ] Step 1: write the options with `Validate()` and its unit tests (red, then green).\n- [ ] Step 2: write `QuestDbTables` DDL builders (use the spec \u00a72.1 *as amended by T0*), with unit tests pinning the\n text for prefix `hist_` with and without TTL.\n- [ ] Step 3: write `QuestDbConnection`.\n - Members:\n - `NpgsqlDataSource` built once;\n - `OpenAsync(ct)`;\n - `CreateCommand(conn, sql)` with `CommandTimeout`;\n - `static bool IsTransient(Exception)`, true for `NpgsqlException` with `IsTransient`, socket and IO errors, and\n timeouts;\n - `static DateTime ReadUtc(NpgsqlDataReader r, int i)`;\n - `string T(string table)`, which returns prefix + table.\n - Use the parameter typing from T0 Q1.\n- [ ] Step 4: write the bootstrapper and fixture. Bring up the overlay\n (`docker compose -f docker-dev/docker-compose.yml -f docker-dev/docker-compose.questdb.yml up -d questdb`), then\n write and run `BootstrapTests`.\n- [ ] Step 5: wire the slnx and the parity-test reference, then build the solution.\n- [ ] Step 6: commit `feat(historian): QuestDB backend skeleton \u2014 options, connection, schema bootstrap with TTL` with\n the trailer.\n\n```json:metadata\n{\"files\":[\"src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/\",\"tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests/\",\"tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/\",\"ZB.MOM.WW.OtOpcUa.slnx\"],\"verifyCommand\":\"dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests --filter FullyQualifiedName~Bootstrap (env set)\",\"acceptanceCriteria\":[\"options validate naming keys\",\"DDL pinned by unit tests\",\"bootstrap idempotent + concurrent + TTL\",\"fixture env-gated with unique prefix\"],\"modelTier\":\"standard\",\"model\":\"sonnet\",\"wave\":2}\n```\n\n---", + "description": "> **Review amendments (Fable) \u2014 binding:**\n> - **Files gain** `QuestDbTagCatalog.cs` (R7) and `QuestDbWalVisibility.cs` (R5, moved from T6).\n> - `QuestDbConnection` gains `Round(DateTime)` and `BindTimestamp` (R6), plus `BindByte` and `ReadKind`, so the T0 Q1\n> answers live in one file (S2).\n> - The DDL uses `num_value`/`int_value`/`str_value` (R9).\n> - **New tests:**\n> - `BootstrapTests.ExistsAsync_is_false_for_an_unknown_tag_and_true_after_EnsureAsync`;\n> - `Wait_returns_once_writerTxn_reaches_the_captured_sequencerTxn` (or the Q4 fallback form);\n> - `Timeout_zero_skips_the_wait`;\n> - a `QuestDb.Tests` unit test pinning that `BindTimestamp` never emits `DateTime.MinValue`/`MaxValue`.\n> - Copy `Platforms AnyCPU;x64` (N4).\n\n\n**Goal:** a building `Driver.Historian.QuestDb` project. It has validated options, a pooled Npgsql connection, an\nidempotent and concurrency-safe schema bootstrapper with TTL, and an env-gated integration fixture.\n\n**Files:**\n- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.csproj`\n (references Core.Abstractions, Core.AlarmHistorian, Server/Runtime and Sql.Common; packages `Npgsql` and\n `Microsoft.Extensions.Logging.Abstractions`; InternalsVisibleTo the two QuestDb test assemblies)\n- Create: `QuestDbHistorianOptions.cs`, `QuestDbConnection.cs`, `QuestDbTables.cs`, `QuestDbSchemaBootstrapper.cs`\n- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests/` (csproj, `QuestDbHistorianOptionsTests.cs`,\n `QuestDbTablesTests.cs`)\n- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/` (csproj with the\n conformance-include line and a ProjectReference to Sql.Common, `QuestDbFixture.cs`, `BootstrapTests.cs`)\n- Modify: `ZB.MOM.WW.OtOpcUa.slnx`, and the Core.Abstractions.Tests csproj (ProjectReference to the QuestDb project)\n\n**Acceptance Criteria:**\n- [ ] `QuestDbHistorianOptions` has `ConnectionString` (default `\"\"`), `TablePrefix` (`\"hist_\"`, regex\n `^[A-Za-z0-9_]+$`), `RetentionDays` (`int?`, \u2265 1), `CommandTimeout` (30 s, > 0), `AutoCreateSchema` (`true`) and\n `VisibilityTimeout` (5 s, \u2265 0). `Validate()` returns error strings that name the key\n `ServerHistorian:QuestDb:`.\n- [ ] `QuestDbTables` produces the three `CREATE TABLE IF NOT EXISTS` statements, as confirmed by T0, for a given prefix\n and retention. Unit tests pin the exact text.\n- [ ] The bootstrapper is idempotent and survives two concurrent instances, using the catch rule from T0 Q7. It\n re-applies a changed TTL. It uses `BootstrapFailureCache` for the 5 s negative cache.\n- [ ] `QuestDbFixture` reads `HISTORIAN_QUESTDB_TEST_CONNECTION` and skips when it is absent. `UniquePrefix()` returns\n `\"t\" + 12 hex + \"_\"`. `DropPrefixAsync(prefix)` drops every table with that prefix, found through `tables()`.\n- [ ] `BootstrapTests` cover:\n - tables created;\n - a second run is a no-op;\n - two concurrent bootstrappers both succeed;\n - TTL is visible in metadata when `RetentionDays=7` and absent when unset;\n - `AutoCreateSchema=false` against a missing table throws `HistorianSchemaMissingException` from Sql.Common.\n\n**Verify:**\n- `dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests` passes.\n- With `HISTORIAN_QUESTDB_TEST_CONNECTION` set, `dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests --filter \"FullyQualifiedName~Bootstrap\"`\n passes.\n- Without the variable, every test skips.\n\n**Steps:**\n- [ ] Step 1: write the options with `Validate()` and its unit tests (red, then green).\n- [ ] Step 2: write `QuestDbTables` DDL builders (use the spec \u00a72.1 *as amended by T0*), with unit tests pinning the\n text for prefix `hist_` with and without TTL.\n- [ ] Step 3: write `QuestDbConnection`.\n - Members:\n - `NpgsqlDataSource` built once;\n - `OpenAsync(ct)`;\n - `CreateCommand(conn, sql)` with `CommandTimeout`;\n - `static bool IsTransient(Exception)`, true for `NpgsqlException` with `IsTransient`, socket and IO errors, and\n timeouts;\n - `static DateTime ReadUtc(NpgsqlDataReader r, int i)`;\n - `string T(string table)`, which returns prefix + table.\n - Use the parameter typing from T0 Q1.\n- [ ] Step 4: write the bootstrapper and fixture. Bring up the overlay\n (`docker compose -f docker-dev/docker-compose.yml -f docker-dev/docker-compose.questdb.yml up -d questdb`), then\n write and run `BootstrapTests`.\n- [ ] Step 5: wire the slnx and the parity-test reference, then build the solution.\n- [ ] Step 6: commit `feat(historian): QuestDB backend skeleton \u2014 options, connection, schema bootstrap with TTL` with\n the trailer.\n\n```json:metadata\n{\"files\":[\"src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/\",\"tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests/\",\"tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/\",\"ZB.MOM.WW.OtOpcUa.slnx\"],\"verifyCommand\":\"dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests --filter FullyQualifiedName~Bootstrap (env set)\",\"acceptanceCriteria\":[\"options validate naming keys\",\"DDL pinned by unit tests\",\"bootstrap idempotent + concurrent + TTL\",\"fixture env-gated with unique prefix\"],\"modelTier\":\"standard\",\"model\":\"sonnet\",\"wave\":2}\n```\n\n---", "blockedBy": [ 1 ] @@ -38,7 +38,7 @@ "id": 4, "subject": "Task 4: QuestDB writes (value writer, tag provisioner, alarm wr", "status": "pending", - "description": "**Goal:** the recorder, provisioning and alarm-history seams write to QuestDB with the same semantics as SQL Server.\n\n**Files:**\n- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianValueWriter.cs`,\n `QuestDbTagProvisioner.cs`, `QuestDbAlarmHistorianWriter.cs`\n- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/WriteTests.cs`,\n `ProvisioningTests.cs`, `AlarmWriterTests.cs`\n\n**Acceptance Criteria:**\n- [ ] The value writer works as follows:\n - It calls `LiveWritePreparer.Prepare(values, utcNow, resolutionTicks: 10)`.\n - It rejects an invalid tag name (`HistorianTagNames.IsValid`) and returns `false`.\n - It reads the existing timestamps for the tag in the batch range, including tombstones, using the T0 Q11 form.\n - It inserts only the rows that are not yet present, in multi-row parameterised `INSERT`s of at most the T0 Q2\n batch size, with `deleted=false` and `server_ts=utcNow()`.\n - It returns `true` with no SQL when every row already existed.\n - Any exception returns `false`. It never throws, and cancellation also returns `false`.\n- [ ] The provisioner upserts `hist_tag` (`ts = 1970-01-01`, data type from `HistorianDataTypes.FromKind`, eng unit,\n description). Both the request field types and the result tally match the SqlServer provisioner's.\n- [ ] The alarm writer works as follows:\n - It uses the same deterministic `event_id` as the SQL Server writer, through `AlarmEventIds.ComputeEventId` from\n Sql.Common.\n - It inserts events through the dedup key `(ts, event_id)`, so a redelivery is idempotent.\n - `Classify` maps an unknown or non-database exception to RetryPlease, and maps permanent errors as the SqlServer\n writer does.\n- [ ] `WriteTests` cover:\n - typed round trip for all six kinds with a non-Good status;\n - a redelivered batch leaves one row and the first value;\n - a tombstoned key is not resurrected by redelivery (seed a tombstone directly);\n - NaN \u21d2 `BadOutOfRange` Empty;\n - a microsecond collision within a batch keeps the first;\n - a stopped port (`Port=1`) returns `false` without throwing.\n- [ ] Each regression test is falsified once.\n\n**Verify:** with `HISTORIAN_QUESTDB_TEST_CONNECTION` set,\n`dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests --filter \"FullyQualifiedName~Write|FullyQualifiedName~Provisioning|FullyQualifiedName~AlarmWriter\"`\npasses.\n\n**Steps:**\n- [ ] Step 1: write `WriteTests` (red).\n- [ ] Step 2: implement the value writer.\n- [ ] Step 3: make `WriteTests` green, then falsify the first-wins, tombstone and NaN tests.\n- [ ] Step 4: do the provisioner, then the alarm writer, each tests-first.\n- [ ] Step 5: commit `feat(historian): QuestDB live writes, provisioning and alarm events` with the trailer.\n\n```json:metadata\n{\"files\":[\"src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianValueWriter.cs\",\"src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbTagProvisioner.cs\",\"src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbAlarmHistorianWriter.cs\",\"tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/WriteTests.cs\",\"tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/ProvisioningTests.cs\",\"tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/AlarmWriterTests.cs\"],\"verifyCommand\":\"dotnet test \u2026QuestDb.IntegrationTests --filter Write|Provisioning|AlarmWriter (env set)\",\"acceptanceCriteria\":[\"insert-if-absent incl tombstones\",\"never throws\",\"event id deterministic + idempotent\",\"falsified\"],\"modelTier\":\"standard\",\"model\":\"sonnet\",\"wave\":3}\n```\n\n---", + "description": "> **Review amendments (Fable) \u2014 binding:**\n> - The writer calls `QuestDbTagCatalog.EnsureAsync` before inserting (R7). After inserting, it calls\n> `QuestDbWalVisibility.WaitAsync`; a timeout does not change the return value (R5).\n> - **Provisioner (S5):** read the existing `hist_tag` rows per chunk (`WHERE name IN (\u2026)`), merge\n> `COALESCE(request, stored)` for `eng_unit` and `description` client-side, then insert. Pin this with the SqlServer\n> `Upsert_creates_then_changes_DataType_on_rerun\u2026` literal.\n> - The alarm writer uses `AlarmEventIds.ToNumericSeverity`, plus the Q3 fallback if VARCHAR dedup was refused.\n> - Every DateTime is bound via `BindTimestamp` (R6).\n\n\n**Goal:** the recorder, provisioning and alarm-history seams write to QuestDB with the same semantics as SQL Server.\n\n**Files:**\n- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianValueWriter.cs`,\n `QuestDbTagProvisioner.cs`, `QuestDbAlarmHistorianWriter.cs`\n- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/WriteTests.cs`,\n `ProvisioningTests.cs`, `AlarmWriterTests.cs`\n\n**Acceptance Criteria:**\n- [ ] The value writer works as follows:\n - It calls `LiveWritePreparer.Prepare(values, utcNow, resolutionTicks: 10)`.\n - It rejects an invalid tag name (`HistorianTagNames.IsValid`) and returns `false`.\n - It reads the existing timestamps for the tag in the batch range, including tombstones, using the T0 Q11 form.\n - It inserts only the rows that are not yet present, in multi-row parameterised `INSERT`s of at most the T0 Q2\n batch size, with `deleted=false` and `server_ts=utcNow()`.\n - It returns `true` with no SQL when every row already existed.\n - Any exception returns `false`. It never throws, and cancellation also returns `false`.\n- [ ] The provisioner upserts `hist_tag` (`ts = 1970-01-01`, data type from `HistorianDataTypes.FromKind`, eng unit,\n description). Both the request field types and the result tally match the SqlServer provisioner's.\n- [ ] The alarm writer works as follows:\n - It uses the same deterministic `event_id` as the SQL Server writer, through `AlarmEventIds.ComputeEventId` from\n Sql.Common.\n - It inserts events through the dedup key `(ts, event_id)`, so a redelivery is idempotent.\n - `Classify` maps an unknown or non-database exception to RetryPlease, and maps permanent errors as the SqlServer\n writer does.\n- [ ] `WriteTests` cover:\n - typed round trip for all six kinds with a non-Good status;\n - a redelivered batch leaves one row and the first value;\n - a tombstoned key is not resurrected by redelivery (seed a tombstone directly);\n - NaN \u21d2 `BadOutOfRange` Empty;\n - a microsecond collision within a batch keeps the first;\n - a stopped port (`Port=1`) returns `false` without throwing.\n- [ ] Each regression test is falsified once.\n\n**Verify:** with `HISTORIAN_QUESTDB_TEST_CONNECTION` set,\n`dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests --filter \"FullyQualifiedName~Write|FullyQualifiedName~Provisioning|FullyQualifiedName~AlarmWriter\"`\npasses.\n\n**Steps:**\n- [ ] Step 1: write `WriteTests` (red).\n- [ ] Step 2: implement the value writer.\n- [ ] Step 3: make `WriteTests` green, then falsify the first-wins, tombstone and NaN tests.\n- [ ] Step 4: do the provisioner, then the alarm writer, each tests-first.\n- [ ] Step 5: commit `feat(historian): QuestDB live writes, provisioning and alarm events` with the trailer.\n\n```json:metadata\n{\"files\":[\"src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianValueWriter.cs\",\"src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbTagProvisioner.cs\",\"src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbAlarmHistorianWriter.cs\",\"tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/WriteTests.cs\",\"tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/ProvisioningTests.cs\",\"tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/AlarmWriterTests.cs\"],\"verifyCommand\":\"dotnet test \u2026QuestDb.IntegrationTests --filter Write|Provisioning|AlarmWriter (env set)\",\"acceptanceCriteria\":[\"insert-if-absent incl tombstones\",\"never throws\",\"event id deterministic + idempotent\",\"falsified\"],\"modelTier\":\"standard\",\"model\":\"sonnet\",\"wave\":3}\n```\n\n---", "blockedBy": [ 3 ] @@ -47,7 +47,7 @@ "id": 5, "subject": "Task 5: QuestDB reads and aggregates", "status": "pending", - "description": "**Goal:** `QuestDbHistorianDataSource` serves Raw, AtTime, Events and Processed. Results are identical to SQL Server's,\nexcept for microsecond rounding.\n\n**Files:**\n- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianDataSource.cs`,\n `QuestDbAggregateQueries.cs`\n- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/ReadTests.cs`\n\n**Acceptance Criteria:**\n- [ ] Every query filters `NOT deleted`. Every row goes through `StoredSample.FromStored`.\n- [ ] **Raw:** forward and reverse, `LIMIT n+1`, and the same paging contract and continuation semantics as the\n SqlServer data source (re-read from the last timestamp, inclusive). Unbounded when `maxValuesPerNode == 0`.\n Unknown tag \u21d2 an empty result. The tag is resolved through the `hist_tag` existence or the sample table, so the\n read never creates a tag.\n- [ ] **AtTime:** the result list matches the input in length and order. Neighbours come from the T0 Q9 form, and\n `AtTimeEvaluator.Evaluate` decides each value.\n- [ ] **Processed:**\n - It uses `BucketPlan.Create` and `AggregateFamilies.FamilyOf`.\n - It runs one statement per node, grouped by `(cast(ts AS LONG)*10 - @startUnixTicks) / @intervalTicks`.\n - It fills the shared `BucketRow` with the same usable/good/bad/non-numeric definitions as SQL Server: usable = status\n severity not Bad and kind \u2260 Empty; good = usable and status < 0x40000000; non-numeric = usable and kind in (4, 5).\n - `AggregateEvaluator.Evaluate` produces the values, and `BucketPlan.Assemble` fills empty buckets and handles\n reversal.\n - An unsupported aggregate throws `HistoryAggregateNotSupportedException`.\n- [ ] **Events:** filtered by source and time range, ascending, limited to `maxEvents`. The `ContinuationPoint` is\n non-null when truncated.\n- [ ] Health counters (`GetHealthSnapshot`) are implemented as in the SqlServer data source (lock-guarded success/failure counters plus the last error). A failure fails only that call and is recorded in\n `GetHealthSnapshot`.\n- [ ] `ReadTests` cover only QuestDB-specific behaviour:\n - tombstoned rows are invisible to Raw, AtTime and Processed (seed a tombstone directly);\n - a whole-range Processed read from 1601 to 9999 with interval 0 does not overflow;\n - a reverse paging walk with `maxValuesPerNode=1` returns every row.\n The shared literals run through the conformance suite in T7.\n\n**Verify:** with `HISTORIAN_QUESTDB_TEST_CONNECTION` set,\n`dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests --filter \"FullyQualifiedName~ReadTests\"`\npasses.\n\n**Steps:**\n- [ ] Step 1: write `ReadTests` (red).\n- [ ] Step 2: implement Raw, then AtTime, Events and Processed.\n- [ ] Step 3: temporarily compile the conformance read classes into this task's worktree. Add a throwaway concrete\n subclass and a throwaway backend that seeds through direct INSERTs. Run the Raw, AtTime and Processed\n conformance classes locally and get them green.\n- [ ] Step 4: delete the throwaway files before committing. T7 owns the real ones. Paste the conformance run output\n into the report.\n- [ ] Step 5: falsify the tombstone filter and the bucket expression (an off-by-one start).\n- [ ] Step 6: commit `feat(historian): QuestDB reads and aggregates (pushed down, shared evaluation)` with the trailer.\n\n```json:metadata\n{\"files\":[\"src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianDataSource.cs\",\"src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbAggregateQueries.cs\",\"tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/ReadTests.cs\"],\"verifyCommand\":\"dotnet test \u2026QuestDb.IntegrationTests --filter FullyQualifiedName~ReadTests (env set) + throwaway conformance run in report\",\"acceptanceCriteria\":[\"NOT deleted everywhere\",\"bucket expr exact vs SqlServer\",\"shared evaluators used\",\"conformance read classes green locally\"],\"modelTier\":\"frontier\",\"model\":\"opus\",\"wave\":3}\n```\n\n---", + "description": "> **Review amendments (Fable) \u2014 binding:**\n> - **Raw per R8:** `LIMIT @top`, at most `top` samples, `ContinuationPoint: null`. Delete the \"LIMIT n+1\" text.\n> - Every inbound timestamp is rounded via `QuestDbConnection.Round`, and every DateTime is bound via `BindTimestamp`\n> (R6). The Processed window is rounded before `BucketPlan.Create`.\n> - Tags are resolved through `QuestDbTagCatalog.ExistsAsync` only (R7).\n> - **SQL classification (S6):** usable \u21d4 `status < {StatusBits.SeverityBadBit}` AND `kind <> 0`; good \u21d4 usable AND\n> `status < {StatusBits.SeverityUncertainBit}`. The literals are interpolated from the consts, never typed by hand. Note\n> that QuestDB's \"not Bad\" is an unsigned LONG comparison, not T-SQL's `>= 0`.\n> - Health uses the shared `HistorianHealthCounters`. Events use `EventContinuation` and\n> `HistoricalEventRows.ClampSeverity`.\n> - **T5 owns `IT/QuestDbConformanceBackend.cs`** (seed and read-back only; N6). Step 3's throwaway backend IS this file,\n> which is kept. Only the throwaway subclasses are deleted.\n> - **Extra test (N2):** `ReadTests` includes one seeded Int64-kind row with `num_value = NULL`, proving the kind, not\n> the column, decides the value.\n\n\n**Goal:** `QuestDbHistorianDataSource` serves Raw, AtTime, Events and Processed. Results are identical to SQL Server's,\nexcept for microsecond rounding.\n\n**Files:**\n- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianDataSource.cs`,\n `QuestDbAggregateQueries.cs`\n- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/ReadTests.cs`\n\n**Acceptance Criteria:**\n- [ ] Every query filters `NOT deleted`. Every row goes through `StoredSample.FromStored`.\n- [ ] **Raw:** forward and reverse, `LIMIT n+1`, and the same paging contract and continuation semantics as the\n SqlServer data source (re-read from the last timestamp, inclusive). Unbounded when `maxValuesPerNode == 0`.\n Unknown tag \u21d2 an empty result. The tag is resolved through the `hist_tag` existence or the sample table, so the\n read never creates a tag.\n- [ ] **AtTime:** the result list matches the input in length and order. Neighbours come from the T0 Q9 form, and\n `AtTimeEvaluator.Evaluate` decides each value.\n- [ ] **Processed:**\n - It uses `BucketPlan.Create` and `AggregateFamilies.FamilyOf`.\n - It runs one statement per node, grouped by `(cast(ts AS LONG)*10 - @startUnixTicks) / @intervalTicks`.\n - It fills the shared `BucketRow` with the same usable/good/bad/non-numeric definitions as SQL Server: usable = status\n severity not Bad and kind \u2260 Empty; good = usable and status < 0x40000000; non-numeric = usable and kind in (4, 5).\n - `AggregateEvaluator.Evaluate` produces the values, and `BucketPlan.Assemble` fills empty buckets and handles\n reversal.\n - An unsupported aggregate throws `HistoryAggregateNotSupportedException`.\n- [ ] **Events:** filtered by source and time range, ascending, limited to `maxEvents`. The `ContinuationPoint` is\n non-null when truncated.\n- [ ] Health counters (`GetHealthSnapshot`) are implemented as in the SqlServer data source (lock-guarded success/failure counters plus the last error). A failure fails only that call and is recorded in\n `GetHealthSnapshot`.\n- [ ] `ReadTests` cover only QuestDB-specific behaviour:\n - tombstoned rows are invisible to Raw, AtTime and Processed (seed a tombstone directly);\n - a whole-range Processed read from 1601 to 9999 with interval 0 does not overflow;\n - a reverse paging walk with `maxValuesPerNode=1` returns every row.\n The shared literals run through the conformance suite in T7.\n\n**Verify:** with `HISTORIAN_QUESTDB_TEST_CONNECTION` set,\n`dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests --filter \"FullyQualifiedName~ReadTests\"`\npasses.\n\n**Steps:**\n- [ ] Step 1: write `ReadTests` (red).\n- [ ] Step 2: implement Raw, then AtTime, Events and Processed.\n- [ ] Step 3: temporarily compile the conformance read classes into this task's worktree. Add a throwaway concrete\n subclass and a throwaway backend that seeds through direct INSERTs. Run the Raw, AtTime and Processed\n conformance classes locally and get them green.\n- [ ] Step 4: delete the throwaway files before committing. T7 owns the real ones. Paste the conformance run output\n into the report.\n- [ ] Step 5: falsify the tombstone filter and the bucket expression (an off-by-one start).\n- [ ] Step 6: commit `feat(historian): QuestDB reads and aggregates (pushed down, shared evaluation)` with the trailer.\n\n```json:metadata\n{\"files\":[\"src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianDataSource.cs\",\"src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbAggregateQueries.cs\",\"tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/ReadTests.cs\"],\"verifyCommand\":\"dotnet test \u2026QuestDb.IntegrationTests --filter FullyQualifiedName~ReadTests (env set) + throwaway conformance run in report\",\"acceptanceCriteria\":[\"NOT deleted everywhere\",\"bucket expr exact vs SqlServer\",\"shared evaluators used\",\"conformance read classes green locally\"],\"modelTier\":\"frontier\",\"model\":\"opus\",\"wave\":3}\n```\n\n---", "blockedBy": [ 3 ] @@ -56,7 +56,7 @@ "id": 6, "subject": "Task 6: QuestDB HistoryUpdate (soft delete) with read-your-writ", "status": "pending", - "description": "**Goal:** `QuestDbHistorianDataUpdater` implements all five operations through insert-only upserts and tombstones. It\nreturns codes identical to SQL Server's, and waits for WAL visibility before returning.\n\n**Files:**\n- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianDataUpdater.cs`,\n `QuestDbWalVisibility.cs`\n- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/UpdateTests.cs`,\n `TombstoneTests.cs`\n\n**Acceptance Criteria:**\n- [ ] `UpdateDataAsync`:\n 1. `HistoryUpdatePlanner.Plan(tag, mode, values, resolutionTicks: 10)` builds the plan.\n 2. It reads which winner timestamps currently have a **live** row, meaning present and not deleted.\n 3. It writes:\n - Insert: only the absent winners;\n - Replace: only the present winners;\n - Update: every winner.\n Each write is an `INSERT` with `deleted=false`.\n 4. It builds the outcome map (present \u21d2 Replaced, absent \u21d2 Inserted, for rows it wrote).\n 5. `HistoryUpdatePlanner.Complete(...)` produces the codes.\n- [ ] `DeleteAtTimeAsync` writes a tombstone for each time with a live row, with `deleted=true`, value columns NULL and\n `status=BadNoData`. Codes come from `HistoryUpdatePlanner.DeleteAtTimeResults`.\n- [ ] `DeleteRawAsync` writes a tombstone for every live row in `[start, end)`. The code comes from\n `HistoryUpdatePlanner.DeleteRawResult`.\n- [ ] On an unknown tag, Insert and Update create the `hist_tag` row; Replace and Delete do not, and return the shared\n unknown-tag codes.\n- [ ] `QuestDbWalVisibility.WaitAsync(table, ct)` works as follows:\n - It polls the T0 Q4 query until the table's applied transaction is at least the sequencer transaction captured\n after our write.\n - It gives up after `VisibilityTimeout` and logs a Warning, without changing the codes. `VisibilityTimeout = 0`\n skips the wait.\n - Every updater operation calls it once, after writing.\n- [ ] `UpdateTests` cover the five operations' codes (QuestDB-specific checks only; the literal codes run through\n conformance in T7) and that a read made right after `UpdateDataAsync` returns sees the change.\n- [ ] `TombstoneTests` cover:\n - delete then Raw read \u21d2 the row is gone;\n - delete, then a `QuestDbHistorianValueWriter` redelivery of the same sample \u21d2 the row stays gone (this needs T4's\n writer: write the test against the interface, and if T4 is not merged yet, seed a tombstone and assert the\n writer's existence query counts it);\n - insert over a tombstone \u21d2 `GoodEntryInserted` and a visible row.\n\n**Verify:** with `HISTORIAN_QUESTDB_TEST_CONNECTION` set,\n`dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests --filter \"FullyQualifiedName~UpdateTests|FullyQualifiedName~TombstoneTests\"`\npasses.\n\n**Steps:**\n- [ ] Step 1: write the tests (red).\n- [ ] Step 2: implement `QuestDbWalVisibility`.\n- [ ] Step 3: implement the updater.\n- [ ] Step 4: get to green. Falsify the \"live row\" filter by dropping `NOT deleted` from the existence check, and the\n visibility wait by setting the timeout to 0 in the read-after-write test.\n- [ ] Step 5: commit `feat(historian): QuestDB HistoryUpdate with soft delete and read-your-writes` with the trailer.\n\n```json:metadata\n{\"files\":[\"src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianDataUpdater.cs\",\"src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbWalVisibility.cs\",\"tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/UpdateTests.cs\",\"tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/TombstoneTests.cs\"],\"verifyCommand\":\"dotnet test \u2026QuestDb.IntegrationTests --filter UpdateTests|TombstoneTests (env set)\",\"acceptanceCriteria\":[\"insert-only upserts, no UPDATE/DELETE\",\"codes from shared planner\",\"tombstones never resurrected\",\"read-your-writes wait\"],\"modelTier\":\"standard\",\"model\":\"sonnet\",\"wave\":3}\n```\n\n---", + "description": "> **Review amendments (Fable) \u2014 binding:**\n> - `QuestDbWalVisibility.cs` moves to T3, so T6 consumes it (R5).\n> - Insert and Update call `QuestDbTagCatalog.EnsureAsync`; Replace and the deletes call `ExistsAsync` (R7).\n> - **Existence read (S15):** one `WHERE tag=$1 AND ts >= $min AND ts <= $max AND NOT deleted` range query, filtered to\n> the winners client-side. Never an `IN` list of `n` parameters.\n> - **DeleteRaw (S12):** read live rows in `LIMIT` pages of the Q2 batch size, keyed on the last `ts`, and tombstone them\n> page by page. `rowsAffected` is the total.\n> - Inbound times are rounded and bound via `BindTimestamp` (R6).\n> - The TombstoneTests \"redelivery after delete\" case moves to T7 as\n> `IT/TombstoneTests.Redelivery.cs::Tombstone_is_not_resurrected_by_a_writer_redelivery` (S8). T6 keeps the other two.\n\n\n**Goal:** `QuestDbHistorianDataUpdater` implements all five operations through insert-only upserts and tombstones. It\nreturns codes identical to SQL Server's, and waits for WAL visibility before returning.\n\n**Files:**\n- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianDataUpdater.cs`,\n `QuestDbWalVisibility.cs`\n- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/UpdateTests.cs`,\n `TombstoneTests.cs`\n\n**Acceptance Criteria:**\n- [ ] `UpdateDataAsync`:\n 1. `HistoryUpdatePlanner.Plan(tag, mode, values, resolutionTicks: 10)` builds the plan.\n 2. It reads which winner timestamps currently have a **live** row, meaning present and not deleted.\n 3. It writes:\n - Insert: only the absent winners;\n - Replace: only the present winners;\n - Update: every winner.\n Each write is an `INSERT` with `deleted=false`.\n 4. It builds the outcome map (present \u21d2 Replaced, absent \u21d2 Inserted, for rows it wrote).\n 5. `HistoryUpdatePlanner.Complete(...)` produces the codes.\n- [ ] `DeleteAtTimeAsync` writes a tombstone for each time with a live row, with `deleted=true`, value columns NULL and\n `status=BadNoData`. Codes come from `HistoryUpdatePlanner.DeleteAtTimeResults`.\n- [ ] `DeleteRawAsync` writes a tombstone for every live row in `[start, end)`. The code comes from\n `HistoryUpdatePlanner.DeleteRawResult`.\n- [ ] On an unknown tag, Insert and Update create the `hist_tag` row; Replace and Delete do not, and return the shared\n unknown-tag codes.\n- [ ] `QuestDbWalVisibility.WaitAsync(table, ct)` works as follows:\n - It polls the T0 Q4 query until the table's applied transaction is at least the sequencer transaction captured\n after our write.\n - It gives up after `VisibilityTimeout` and logs a Warning, without changing the codes. `VisibilityTimeout = 0`\n skips the wait.\n - Every updater operation calls it once, after writing.\n- [ ] `UpdateTests` cover the five operations' codes (QuestDB-specific checks only; the literal codes run through\n conformance in T7) and that a read made right after `UpdateDataAsync` returns sees the change.\n- [ ] `TombstoneTests` cover:\n - delete then Raw read \u21d2 the row is gone;\n - delete, then a `QuestDbHistorianValueWriter` redelivery of the same sample \u21d2 the row stays gone (this needs T4's\n writer: write the test against the interface, and if T4 is not merged yet, seed a tombstone and assert the\n writer's existence query counts it);\n - insert over a tombstone \u21d2 `GoodEntryInserted` and a visible row.\n\n**Verify:** with `HISTORIAN_QUESTDB_TEST_CONNECTION` set,\n`dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests --filter \"FullyQualifiedName~UpdateTests|FullyQualifiedName~TombstoneTests\"`\npasses.\n\n**Steps:**\n- [ ] Step 1: write the tests (red).\n- [ ] Step 2: implement `QuestDbWalVisibility`.\n- [ ] Step 3: implement the updater.\n- [ ] Step 4: get to green. Falsify the \"live row\" filter by dropping `NOT deleted` from the existence check, and the\n visibility wait by setting the timeout to 0 in the read-after-write test.\n- [ ] Step 5: commit `feat(historian): QuestDB HistoryUpdate with soft delete and read-your-writes` with the trailer.\n\n```json:metadata\n{\"files\":[\"src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianDataUpdater.cs\",\"src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbWalVisibility.cs\",\"tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/UpdateTests.cs\",\"tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/TombstoneTests.cs\"],\"verifyCommand\":\"dotnet test \u2026QuestDb.IntegrationTests --filter UpdateTests|TombstoneTests (env set)\",\"acceptanceCriteria\":[\"insert-only upserts, no UPDATE/DELETE\",\"codes from shared planner\",\"tombstones never resurrected\",\"read-your-writes wait\"],\"modelTier\":\"standard\",\"model\":\"sonnet\",\"wave\":3}\n```\n\n---", "blockedBy": [ 3 ] @@ -65,7 +65,7 @@ "id": 7, "subject": "Task 7: Provider, Host wiring and the conformance suite on Ques", "status": "pending", - "description": "**Goal:** `Backend=QuestDb` resolves, validates and builds all five seams. The full conformance suite passes against\nQuestDB.\n\n**Files:**\n- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianProvider.cs`\n- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/QuestDbConformanceBackend.cs`,\n `QuestDbConformance.cs` (one concrete subclass per conformance class)\n- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests/QuestDbHistorianProviderTests.cs`\n- Modify:\n - `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/HistorianBackendRegistry.cs` (add a `QuestDbBackendName`\n const, `BindQuestDb(IConfiguration)` and a `Factories` entry);\n - `ServerHistorianOptionsValidator.cs` (a QuestDb block before the `if (!options.IsGatewayBackend) return;` line,\n mirroring the SqlServer block);\n - `src/Server/ZB.MOM.WW.OtOpcUa.Host/ZB.MOM.WW.OtOpcUa.Host.csproj` (a ProjectReference);\n - `tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests/Configuration/HistorianBackendRegistryTests.cs`;\n - `tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/ServerHistorianOptionsValidatorTests.cs`;\n - `docker-dev/docker-compose.yml`: in the central-1 and central-2 `ServerHistorian__*` blocks, add\n `ServerHistorian__QuestDb__ConnectionString: \"${OTOPCUA_HISTORIAN_QUESTDB:-}\"`\n plus a comment line naming the overlay.\n\n**Acceptance Criteria:**\n- [ ] The provider has `Name=\"QuestDb\"`, `SupportedKinds` = all six kinds as a FrozenSet, and one lazily built shared\n connection and bootstrapper across the seams. The constructor does no I/O and no validation, like the SqlServer\n provider. `CreateUpdater` returns the soft-delete updater.\n- [ ] The registry resolves `QuestDb` case-insensitively.\n- [ ] The validator, when the backend is consumed and is `QuestDb`:\n - reports every `QuestDbHistorianOptions.Validate()` error, naming the key;\n - does not require the Gateway keys;\n - ignores an invalid `TablePrefix` when not consumed.\n- [ ] Registry and validator tests mirror the five SqlServer ones (lines 71 and 319\u2013368).\n- [ ] The QuestDB integration project runs the **entire** conformance suite through `QuestDbConformanceBackend`:\n - `ResolutionTicks = 10`;\n - seeding is direct INSERTs into the prefixed tables;\n - `ReadBackAsync` filters `NOT deleted`.\n Every class passes. Any named override has a doc comment and a ledger ruling.\n\n**Verify:**\n- With both environment variables set, the SqlServer and QuestDB integration projects both pass. The report shows the\n conformance class counts side by side.\n- `dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests --filter \"FullyQualifiedName~Historian\"` passes.\n- `dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests --filter \"FullyQualifiedName~Historian\"` passes.\n\n**Steps:**\n- [ ] Step 1: write the provider unit tests (red), then the provider.\n- [ ] Step 2: write the registry and validator tests (red), then the wiring.\n- [ ] Step 3: write `QuestDbConformanceBackend` and the subclasses, and run the full suite. Fix any divergence in the\n QuestDB code, even though those files belong to T4\u2013T6; this task is the integration owner in W4.\n- [ ] Step 4: make the compose env edit.\n- [ ] Step 5: build the solution, then commit `feat(historian): QuestDb provider + Host wiring; conformance suite green\n on both backends` with the trailer.\n\n```json:metadata\n{\"files\":[\"src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianProvider.cs\",\"tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/QuestDbConformanceBackend.cs\",\"tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/QuestDbConformance.cs\",\"src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/HistorianBackendRegistry.cs\",\"src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/ServerHistorianOptionsValidator.cs\",\"src/Server/ZB.MOM.WW.OtOpcUa.Host/ZB.MOM.WW.OtOpcUa.Host.csproj\",\"docker-dev/docker-compose.yml\"],\"verifyCommand\":\"both IT projects green with env set; Host historian filters green\",\"acceptanceCriteria\":[\"provider shape\",\"registry + consumer-gated validator\",\"full conformance suite green on QuestDB\",\"compose env\"],\"modelTier\":\"standard\",\"model\":\"sonnet\",\"wave\":4}\n```\n\n---", + "description": "> **Review amendments (Fable) \u2014 binding:**\n> - The model is **opus** (R10). T7 may fix T4/T6 files and the data-source Raw/Events plumbing. Aggregate and AtTime\n> divergences are reported to the controller, not fixed.\n> - T5 owns `QuestDbConformanceBackend.cs`; T7 owns the subclasses (N6).\n> - T7 adds `IT/TombstoneTests.Redelivery.cs` (S8).\n> - **Compose default (S14):** the exact Q1-confirmed connection string, with `Host=questdb;Port=8812` substituted,\n> keeping every keyword T0 found necessary.\n\n\n**Goal:** `Backend=QuestDb` resolves, validates and builds all five seams. The full conformance suite passes against\nQuestDB.\n\n**Files:**\n- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianProvider.cs`\n- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/QuestDbConformanceBackend.cs`,\n `QuestDbConformance.cs` (one concrete subclass per conformance class)\n- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests/QuestDbHistorianProviderTests.cs`\n- Modify:\n - `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/HistorianBackendRegistry.cs` (add a `QuestDbBackendName`\n const, `BindQuestDb(IConfiguration)` and a `Factories` entry);\n - `ServerHistorianOptionsValidator.cs` (a QuestDb block before the `if (!options.IsGatewayBackend) return;` line,\n mirroring the SqlServer block);\n - `src/Server/ZB.MOM.WW.OtOpcUa.Host/ZB.MOM.WW.OtOpcUa.Host.csproj` (a ProjectReference);\n - `tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests/Configuration/HistorianBackendRegistryTests.cs`;\n - `tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/ServerHistorianOptionsValidatorTests.cs`;\n - `docker-dev/docker-compose.yml`: in the central-1 and central-2 `ServerHistorian__*` blocks, add\n `ServerHistorian__QuestDb__ConnectionString: \"${OTOPCUA_HISTORIAN_QUESTDB:-}\"`\n plus a comment line naming the overlay.\n\n**Acceptance Criteria:**\n- [ ] The provider has `Name=\"QuestDb\"`, `SupportedKinds` = all six kinds as a FrozenSet, and one lazily built shared\n connection and bootstrapper across the seams. The constructor does no I/O and no validation, like the SqlServer\n provider. `CreateUpdater` returns the soft-delete updater.\n- [ ] The registry resolves `QuestDb` case-insensitively.\n- [ ] The validator, when the backend is consumed and is `QuestDb`:\n - reports every `QuestDbHistorianOptions.Validate()` error, naming the key;\n - does not require the Gateway keys;\n - ignores an invalid `TablePrefix` when not consumed.\n- [ ] Registry and validator tests mirror the five SqlServer ones (lines 71 and 319\u2013368).\n- [ ] The QuestDB integration project runs the **entire** conformance suite through `QuestDbConformanceBackend`:\n - `ResolutionTicks = 10`;\n - seeding is direct INSERTs into the prefixed tables;\n - `ReadBackAsync` filters `NOT deleted`.\n Every class passes. Any named override has a doc comment and a ledger ruling.\n\n**Verify:**\n- With both environment variables set, the SqlServer and QuestDB integration projects both pass. The report shows the\n conformance class counts side by side.\n- `dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests --filter \"FullyQualifiedName~Historian\"` passes.\n- `dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests --filter \"FullyQualifiedName~Historian\"` passes.\n\n**Steps:**\n- [ ] Step 1: write the provider unit tests (red), then the provider.\n- [ ] Step 2: write the registry and validator tests (red), then the wiring.\n- [ ] Step 3: write `QuestDbConformanceBackend` and the subclasses, and run the full suite. Fix any divergence in the\n QuestDB code, even though those files belong to T4\u2013T6; this task is the integration owner in W4.\n- [ ] Step 4: make the compose env edit.\n- [ ] Step 5: build the solution, then commit `feat(historian): QuestDb provider + Host wiring; conformance suite green\n on both backends` with the trailer.\n\n```json:metadata\n{\"files\":[\"src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianProvider.cs\",\"tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/QuestDbConformanceBackend.cs\",\"tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/QuestDbConformance.cs\",\"src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/HistorianBackendRegistry.cs\",\"src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/ServerHistorianOptionsValidator.cs\",\"src/Server/ZB.MOM.WW.OtOpcUa.Host/ZB.MOM.WW.OtOpcUa.Host.csproj\",\"docker-dev/docker-compose.yml\"],\"verifyCommand\":\"both IT projects green with env set; Host historian filters green\",\"acceptanceCriteria\":[\"provider shape\",\"registry + consumer-gated validator\",\"full conformance suite green on QuestDB\",\"compose env\"],\"modelTier\":\"frontier\",\"model\":\"opus\",\"wave\":4}\n```\n\n---", "blockedBy": [ 2, 4, @@ -77,7 +77,7 @@ "id": 8, "subject": "Task 8: Documentation", "status": "pending", - "description": "**Goal:** the docs describe the QuestDB backend as built, and every claim is checked against the merged code.\n\n**Files:** `docs/Historian.md` (a new \u00a7\"QuestDB backend\"), `docs/Configuration.md` (the `ServerHistorian:QuestDb`\ntable), `CLAUDE.md` (one short bullet in the Pluggable backends subsection), `docker-dev/README.md` (the overlay\nrecipe), and the spec (an as-built pass)\n\n**Acceptance Criteria:**\n- [ ] Every `ServerHistorian:QuestDb` key is documented with its default and validation rule.\n- [ ] The QuestDB-specific behaviour is documented:\n - soft delete and tombstones;\n - insert-if-absent including tombstones;\n - microsecond truncation and collisions;\n - WAL visibility and the read-your-writes wait;\n - TTL;\n - the overlay recipe and `HISTORIAN_QUESTDB_TEST_CONNECTION`.\n- [ ] Each statement is cross-checked against the code, and any spec-versus-code disagreement is fixed in the spec\n inline and listed in the report.\n\n**Verify:** the controller reads the diff, and `grep -n \"QuestDb\" docs/Configuration.md` shows the table.\n\n```json:metadata\n{\"files\":[\"docs/Historian.md\",\"docs/Configuration.md\",\"CLAUDE.md\",\"docker-dev/README.md\",\"docs/plans/2026-10-10-questdb-historian-design.md\"],\"verifyCommand\":\"grep -n QuestDb docs/Configuration.md\",\"acceptanceCriteria\":[\"keys documented\",\"QuestDB-specific semantics documented\",\"claims cross-checked\"],\"modelTier\":\"standard\",\"model\":\"sonnet\",\"wave\":5}\n```\n\n---", + "description": "> **Review amendments (Fable) \u2014 binding:**\n> - Document these, in addition to the existing list:\n> - R6's sub-\u00b5s request-start consequence;\n> - QuestDB `LONG` NULL = `long.MinValue`, so a stored `FromInt64(long.MinValue)` reads back Empty (N1);\n> - TTL is enforced on write commits only, so a quiet table never drops (N3);\n> - all write paths wait for WAL visibility (R5).\n> - **CLAUDE.md (N5):** check the Historian section's opening paragraph. If it still says\n> \"HistorianGateway (sole historian backend)\", rewrite it to name all three backends. Otherwise add QuestDb to the\n> existing \"Pluggable backends\" subsection.\n\n\n**Goal:** the docs describe the QuestDB backend as built, and every claim is checked against the merged code.\n\n**Files:** `docs/Historian.md` (a new \u00a7\"QuestDB backend\"), `docs/Configuration.md` (the `ServerHistorian:QuestDb`\ntable), `CLAUDE.md` (one short bullet in the Pluggable backends subsection), `docker-dev/README.md` (the overlay\nrecipe), and the spec (an as-built pass)\n\n**Acceptance Criteria:**\n- [ ] Every `ServerHistorian:QuestDb` key is documented with its default and validation rule.\n- [ ] The QuestDB-specific behaviour is documented:\n - soft delete and tombstones;\n - insert-if-absent including tombstones;\n - microsecond truncation and collisions;\n - WAL visibility and the read-your-writes wait;\n - TTL;\n - the overlay recipe and `HISTORIAN_QUESTDB_TEST_CONNECTION`.\n- [ ] Each statement is cross-checked against the code, and any spec-versus-code disagreement is fixed in the spec\n inline and listed in the report.\n\n**Verify:** the controller reads the diff, and `grep -n \"QuestDb\" docs/Configuration.md` shows the table.\n\n```json:metadata\n{\"files\":[\"docs/Historian.md\",\"docs/Configuration.md\",\"CLAUDE.md\",\"docker-dev/README.md\",\"docs/plans/2026-10-10-questdb-historian-design.md\"],\"verifyCommand\":\"grep -n QuestDb docs/Configuration.md\",\"acceptanceCriteria\":[\"keys documented\",\"QuestDB-specific semantics documented\",\"claims cross-checked\"],\"modelTier\":\"standard\",\"model\":\"sonnet\",\"wave\":5}\n```\n\n---", "blockedBy": [ 7 ] @@ -92,5 +92,5 @@ ] } ], - "lastUpdated": "2026-10-10T12:55:44.872528Z" + "lastUpdated": "2026-10-10T13:09:42.579032+00:00" } \ No newline at end of file From cdea173bd10fab458e33b43649264d44ddf94071 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Sun, 11 Oct 2026 01:07:38 -0400 Subject: [PATCH 04/23] =?UTF-8?q?spike(historian):=20QuestDB=20behaviours?= =?UTF-8?q?=20confirmed=20against=20pinned=20image;=20spec=20=C2=A72?= =?UTF-8?q?=E2=80=93=C2=A74=20amended?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit questdb/questdb:10.0.1 (pushed 2026-08-24). Q1–Q13 answered with harness output in docs/plans/2026-10-10-questdb-spike.md; confirmed DDL, visibility wait, bucket/neighbour SQL and connection keywords folded into the design spec, marked T0-confirmed. Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe --- docker-dev/docker-compose.questdb.yml | 28 ++ .../2026-10-10-questdb-historian-design.md | 188 +++++++- docs/plans/2026-10-10-questdb-spike.md | 411 ++++++++++++++++++ 3 files changed, 608 insertions(+), 19 deletions(-) create mode 100644 docker-dev/docker-compose.questdb.yml create mode 100644 docs/plans/2026-10-10-questdb-spike.md diff --git a/docker-dev/docker-compose.questdb.yml b/docker-dev/docker-compose.questdb.yml new file mode 100644 index 00000000..a86422c9 --- /dev/null +++ b/docker-dev/docker-compose.questdb.yml @@ -0,0 +1,28 @@ +# docker-dev/docker-compose.questdb.yml — optional QuestDB fixture for the QuestDB historian backend (#666). +# +# Up (touches ONLY the questdb service; it has no depends_on, and --no-deps makes that explicit): +# docker compose -f docker-dev/docker-compose.yml -f docker-dev/docker-compose.questdb.yml up -d --no-deps questdb +# +# Image pinned by the T0 spike (docs/plans/2026-10-10-questdb-spike.md): questdb/questdb:10.0.1, +# pushed to Docker Hub 2026-08-24 (≥ 2 weeks before 2026-10-10; ≥ 8.2.3, the first release with table TTL). +# The image ships /usr/bin/curl, so the healthcheck uses it against the min-HTTP health endpoint on :9003. +# +# PG wire -> localhost:18812 (Host=localhost;Port=18812;Username=admin;Password=quest;Database=qdb;...) +# Console -> http://localhost:19000 +services: + questdb: + image: questdb/questdb:10.0.1 + container_name: otopcua-dev-questdb-1 + ports: + - "18812:8812" # PG wire + - "19000:9000" # web console / REST + volumes: + - otopcua-questdb-data:/var/lib/questdb + healthcheck: + test: ["CMD-SHELL", "curl -fs http://localhost:9003/status || exit 1"] + interval: 10s + timeout: 5s + retries: 12 +volumes: + otopcua-questdb-data: + name: otopcua-questdb-data diff --git a/docs/plans/2026-10-10-questdb-historian-design.md b/docs/plans/2026-10-10-questdb-historian-design.md index 17574b8c..63ec36ed 100644 --- a/docs/plans/2026-10-10-questdb-historian-design.md +++ b/docs/plans/2026-10-10-questdb-historian-design.md @@ -81,28 +81,64 @@ Host changes: QuestDB has no schemas, so the tables carry a configurable `TablePrefix` (default `hist_`, `[A-Za-z0-9_]+`). All three are **WAL tables with `DEDUP UPSERT KEYS`**, so re-sending a row with the same key replaces it. +**T0-confirmed** against `questdb/questdb:10.0.1` (`docs/plans/2026-10-10-questdb-spike.md` §DDL, Q3, Q5, Q10). The +`TTL n DAYS` clause goes **between `PARTITION BY` and `WAL`**. Placed after the dedup clause it fails with `unexpected +token [TTL]`. It is emitted only when `RetentionDays` is set, as an inlined integer, because it cannot be a parameter. + ```sql CREATE TABLE IF NOT EXISTS hist_sample ( ts TIMESTAMP, tag SYMBOL, kind BYTE, num_value DOUBLE, int_value LONG, str_value VARCHAR, status LONG, deleted BOOLEAN, server_ts TIMESTAMP -) TIMESTAMP(ts) PARTITION BY DAY WAL DEDUP UPSERT KEYS(ts, tag) [TTL n DAYS]; +) TIMESTAMP(ts) PARTITION BY DAY [TTL n DAYS] WAL DEDUP UPSERT KEYS(ts, tag); CREATE TABLE IF NOT EXISTS hist_event ( - ts TIMESTAMP, event_id VARCHAR, source SYMBOL, - -- plus the remaining hist.Event columns one-for-one (docs/historian/sqlserver/001_initial.sql), QuestDB-typed -) TIMESTAMP(ts) PARTITION BY DAY WAL DEDUP UPSERT KEYS(ts, event_id) [TTL n DAYS]; + ts TIMESTAMP, event_id VARCHAR, source SYMBOL, alarm_id VARCHAR, alarm_name VARCHAR, alarm_type_name VARCHAR, + severity INT, event_kind VARCHAR, message VARCHAR, user_name VARCHAR, comment VARCHAR, received_utc TIMESTAMP +) TIMESTAMP(ts) PARTITION BY DAY [TTL n DAYS] WAL DEDUP UPSERT KEYS(ts, event_id); CREATE TABLE IF NOT EXISTS hist_tag ( ts TIMESTAMP, name SYMBOL, data_type BYTE, eng_unit VARCHAR, description VARCHAR, updated_utc TIMESTAMP ) TIMESTAMP(ts) PARTITION BY YEAR WAL DEDUP UPSERT KEYS(ts, name); -- ts is always 1970-01-01, which makes this an upsert by name ``` -Column details: -- `status` holds the uint status code as a LONG. -- `int` also holds Booleans as 0/1, matching SQL Server. -- `kind` is `HistorianValueKind`. -- The exact column types and the TTL syntax are confirmed by T0 against the pinned image. T0 amends this section - with what it finds. +`hist_event` maps `hist.Event` one-for-one: +- `EventTimeUtc` → `ts`, `EventId` → `event_id`, `SourceName` → `source`; +- `AlarmId`/`AlarmName`/`AlarmTypeName` → `alarm_id`/`alarm_name`/`alarm_type_name`; +- `Severity` → `severity`, `EventKind` → `event_kind`, `Message` → `message`; +- `UserName` → `user_name`, `Comment` → `comment`, `ReceivedTimeUtc` → `received_utc`. + +`event_id VARCHAR` works as a dedup key (T0 Q3), so re-sending an event replaces it. The S4 fallback (no dedup on +`hist_event`) is not used. + +Column details (**T0-confirmed**): +- `status` holds the uint status code **unsigned** in a LONG (0..4294967295, never negative). So "severity Bad" is + `status >= 2147483648`, not `< 0` as in SQL Server's signed `int`. +- `int_value` also holds Booleans as 0/1, matching SQL Server. +- `kind` is `HistorianValueKind` in a `BYTE`. +- Read-back CLR types: + + | Column type | Npgsql type name | CLR type / reader call | + |---|---|---| + | `BYTE` | `smallint` | `Int16` (`GetInt16`) | + | `SYMBOL`, `VARCHAR` | `character varying` | `String` | + | `LONG` | `bigint` | `Int64` | + | `INT` | `integer` | `Int32` | + | `BOOLEAN` | `boolean` | `Boolean` | + | `DOUBLE` | `double precision` | `Double` | + | `TIMESTAMP` | `timestamp without time zone` | `DateTime`, **`Kind=Unspecified`**; the reader applies `SpecifyKind(…, Utc)` | + +- **NULL semantics:** + - DOUBLE NaN and ±Inf read back as NULL. + - `long.MinValue` (LONG) and `int.MinValue` (INT) are QuestDB's NULL sentinels. + - `BYTE` and `BOOLEAN` have **no NULL**: unset or DBNull reads `0` / `false`. So `deleted` defaults to `false`, and a + LEFT-JOINed "no row" yields `kind = 0`. Absence must be detected on a NULL `ts`, not on `kind`. +- **Ruling candidate (T0):** an Int64 sample of exactly `long.MinValue` stores `int_value` as NULL. The QuestDB read + path repairs it before the shared `FromColumns`: `kind = Int64 ∧ int_value IS NULL ∧ num_value = -2^63` ⇒ + `long.MinValue`. That double is exact. Without the repair the sample would be served as 0. +- A designated timestamp **before 1970-01-01 cannot be inserted** (`designated timestamp before 1970-01-01 is not + allowed`). See §2.2. +- Tag names: SYMBOL and VARCHAR take 400 characters (non-ASCII included). SYMBOL grows past its capacity hint + (16,384) with no error, and was tested with 30,000 distinct tags. No `CAPACITY` clause is needed. ### 2.2 Live writes (recorder) — insert only if absent, the same as SQL Server @@ -110,16 +146,45 @@ Column details: to microseconds before this step, so microsecond collisions also collapse. 2. For each tag, fetch the timestamps that already exist in the batch's `[min ts, max ts]` range. **Tombstones are included**, so a re-sent outbox batch can never resurrect a deleted row or overwrite a HistoryUpdate change. -3. Insert only the new rows, in one parameterised multi-row `INSERT` per tag group. + T0-confirmed form (Q11): `SELECT ts FROM {p}sample WHERE tag = $1 AND ts >= $2 AND ts <= $3`, with no `NOT + deleted`, followed by a client-side set test. +3. Insert only the new rows with parameterised multi-row `INSERT … VALUES (…),(…)`, **at most 1,000 rows per + statement** (T0-confirmed Q2). QuestDB reads the Bind parameter count as a signed int16, so the hard limit is + 32,767 parameters (3,640 nine-column rows). Within one statement a duplicate key keeps the **last** row, which is + why step 1's "first wins" is done client-side. 4. Stored-value rules are shared with SQL Server: - NaN/±Inf, or any Empty with a non-Bad status, is stored Empty with `BadOutOfRange` or `BadNoData`. - Comms-loss outage markers are stored as given. - A tag name over 400 characters is a per-tag failure. + - **Ruling candidate (T0 Q13):** a sample timestamped before 1970-01-01 cannot be stored, because QuestDB rejects + the whole INSERT. The writer drops such samples before the INSERT and logs one Warning per batch. A retry could + never succeed, and sending the sample would push the tag's whole group into the #538 quarantine. 5. Any SQL or transport failure makes the write return `false`, so the recorder retries and the #538 per-tag quarantine still applies. It never throws. The recorder is Primary-only (#537), so step 2 does not race another live writer. The recorder waits for WAL -visibility, bounded by `VisibilityTimeout` (plan R5). +visibility, bounded by `VisibilityTimeout` (plan R5), using the §2.3 wait. + +Binding (T0-confirmed Q1/Q6/Q13): +- Every timestamp parameter is `NpgsqlDbType.TimestampTz` with `DateTimeKind.Utc`, after `Round`. `Timestamp` + `Utc` + throws in Npgsql. Npgsql truncates sub-µs ticks **toward 2000-01-01**, which rounds *up* before 2000. A raw + `DateTime.MinValue`/`MaxValue` is sent as ±infinity, and a `[Min, Max)` range then silently matches nothing. R6's + `BindTimestamp` (round; `MinValue` → `+10` ticks; `MaxValue` → `−9` ticks) fixes all three. +- The other parameter types: + + | Column | `NpgsqlDbType` | + |---|---| + | `tag`/`str_value`/other strings | `Varchar` | + | `kind`/`data_type` | `Smallint` | + | `num_value` | `Double` | + | `int_value`/`status` | `Bigint` | + | `deleted` | `Boolean` | + | `severity` | `Integer` | + + NULL is `DBNull.Value` with the column's type. +- Every QuestDB server error carries **SQLSTATE `00000`**. `QuestDbConnection`'s classification therefore treats a + `PostgresException` as a server-side rejection, judged by `MessageText`. Any other `NpgsqlException`, `IOException`, + `SocketException` or timeout is transport (transient). ### 2.3 HistoryUpdate (soft delete) @@ -140,6 +205,25 @@ Further rules: - **Read-your-writes.** After its inserts, the updater polls `wal_tables()` until the table's applied transaction reaches the sequencer transaction it produced, up to `VisibilityTimeout`. On a timeout the write is kept, the codes are reported as normal, and a Warning is logged. +- **The visibility wait (`QuestDbWalVisibility`, T0-confirmed Q4)** is used by the writer and the updater (R5): + 1. After the INSERT(s) return: `SELECT sequencerTxn FROM wal_tables() WHERE name = $1` (`$1` = the full table name, + `Varchar`) → `target`. Each autocommitted INSERT is exactly one sequencer transaction. + 2. Poll `SELECT writerTxn, suspended FROM wal_tables() WHERE name = $1` about every 10 ms until `writerTxn >= + target`. Stop early, with a Warning, if `suspended` is true. Stop at `VisibilityTimeout`. + - Measured: median 0 ms, p95 6 ms, max 13 ms. Up to 7/20 inserts were not visible to an immediate re-read, so the + wait is required. A concurrent writer can only raise `target`. The key-poll fallback (S11) is not used. +- Existence checks for sparse item times (Insert/Replace/Update/DeleteAtTime) use `SELECT ts, deleted FROM {p}sample + WHERE tag = $1 AND ts IN ($2, …)`, chunked at 1,000 times (T0 Q11). +- **DeleteRaw (T0-confirmed), page size 1,000:** + 1. `SELECT ts FROM {p}sample WHERE tag = $1 AND ts >= $2 AND ts < $3 AND NOT deleted ORDER BY ts LIMIT 1000`. + 2. Write one 1,000-row tombstone INSERT. + 3. Run the visibility wait. + 4. Continue from `last ts + 10 ticks`, until a page comes back empty. + The wait between pages is required, or the next page re-reads rows just tombstoned. Measured: 2,500 rows, 3 pages, + 103 ms. +- **Ruling candidate (T0 Q13):** an Insert, Replace or Update item timestamped before 1970-01-01 returns + `BadOutOfRange`, because QuestDB cannot store it. Deletes there find nothing and return `BadNoEntryExists`. Pre-1970 + conformance seeds are skipped for QuestDB. - The existence checks for Insert and Replace read committed data. If a live recorder sample at the *same* timestamp is still in flight in the WAL, it can race the updater. Updates normally target past timestamps, so this is accepted and documented. @@ -150,20 +234,56 @@ When `RetentionDays` is set, the bootstrapper applies TTL to `hist_sample` and ` the configured value changes. QuestDB then drops expired day partitions, tombstones included. When unset, data is kept forever. `hist_tag` never expires. +T0-confirmed (Q5): +- **Create:** `… PARTITION BY DAY TTL n DAYS WAL …` (§2.1). +- **Change:** `ALTER TABLE {p}sample SET TTL n DAYS`. **Remove** (`RetentionDays` unset while the table has a TTL): + `ALTER TABLE … SET TTL 0 DAYS`. +- `n` cannot be a `$` parameter (`invalid syntax, should be but was $1`). It is the validated integer, + formatted with `InvariantCulture`. +- TTL is not allowed on the `PARTITION BY YEAR` `hist_tag` table (`TTL value must be an integer multiple of the + partition size`). +- **Reading the current TTL:** `SELECT ttlValue, ttlUnit FROM tables() WHERE table_name = $1`. QuestDB **normalises + the unit**: 7 days reads `1 WEEK`, 14 days `2 WEEK`, 30 days `30 DAY`, none `0 HOUR`. The bootstrapper compares in + hours (`HOUR`=1, `DAY`=24, `WEEK`=168; any other unit counts as different) and issues an `ALTER` only on a + difference. +- `ALTER … SET TTL` is applied through the WAL, so `tables()` shows the old value for 25–100 ms. The bootstrapper does + not re-read to verify. +- Expiry is checked against the **wall clock**: the image default is `cairo.ttl.use.wall.clock=true`. A future-dated + sample does not expire current data (T0 Q5c). +- **Table listing:** `SELECT table_name FROM tables() WHERE starts_with(table_name, $1)`. Never `LIKE 'hist_%'`, + because `_` is a wildcard. +- **Concurrent bootstrap:** `CREATE TABLE IF NOT EXISTS` from 4 connections × 20 rounds produced no errors. The S11 + defence stays: catch a `PostgresException` whose `MessageText` contains `already exists`, re-check `tables()`, and + rethrow if the table is absent. Never key on SQLSTATE (always `00000`). + ## 3. Reads Every query filters `NOT deleted`. Every value goes through `StoredSample`, so the quality clamp is the same as SQL Server's. - **Raw.** A range on `ts`, `ORDER BY ts [DESC]`, `LIMIT @top` (`maxValuesPerNode == 0` ⇒ unbounded), at most `top` samples, `ContinuationPoint: null`; the node manager pages by re-reading from the last timestamp (plan R8). The dedup key forbids same-instant ties. Bounds follow the existing data-source contract. -- **AtTime.** Find the previous-or-equal neighbour and the next neighbour of each requested time. The default is - `LIMIT 1` seeks on `ts`, batched into chunked statements; T0 picks `ASOF JOIN` for the previous neighbour if it - measures better. `AtTimeEvaluator` then decides the result. +- **AtTime (T0-confirmed Q9).** Find the previous-or-equal neighbour and the next neighbour of each requested time + with `LIMIT 1` seeks, `UNION ALL`-ed, **at most 50 request times (100 branches) per statement**. QuestDB rejects + 128 or more branches with `SQL model is too complex to evaluate`. Per time `k`, with `$1` = tag and `$j` = the + rounded time: + ```sql + SELECT * FROM (SELECT k AS req, 0 AS dir, ts, kind, num_value, int_value, str_value, status FROM {p}sample + WHERE tag = $1 AND ts <= $j AND NOT deleted ORDER BY ts DESC LIMIT 1) + UNION ALL + SELECT * FROM (SELECT k AS req, 1 AS dir, ts, kind, num_value, int_value, str_value, status FROM {p}sample + WHERE tag = $1 AND ts > $j AND NOT deleted ORDER BY ts ASC LIMIT 1) + ``` + `k` and `dir` are generated integer literals. Measured: 102 times in about 16–20 ms over 100k rows per tag, with no + mismatches. `ASOF JOIN` was faster (1 ms), but it yields only the previous neighbour, so it was not chosen. + `AtTimeEvaluator` then decides the result. - **Events.** Filtered by source and `ts` range, ordered ascending (reversed windows are swapped, as built today), and limited to `maxEvents`. -- **Processed.** **One query per node.** - - Bucket index: `(cast(ts AS LONG) - @startMicros) / @intervalMicros`, using microsecond values from `BucketPlan`, - grouped by the index. +- **Processed.** **One query per node** (T0-confirmed for every family; the S11 two-statement fallback is not needed). + - Bucket index (T0-confirmed): `(cast(ts AS LONG) * 10 - $1) / $2`. + - `cast(ts AS LONG)` is µs since 1970. + - `$1` = the rounded window start in **ticks since the Unix epoch** (`start.Ticks − DateTime.UnixEpoch.Ticks`). + - `$2` = the interval in ticks. Both are `NpgsqlDbType.Bigint`. + - The result is exact because both sides are µs-aligned (R6), and it cannot overflow before year 9999. - The query returns the full `BucketRow` statistics. Usable, good, bad and uncertain counts come from `sum(CASE …)` over the status column. - It does **not** use `SAMPLE BY`, whose alignment rules differ from ours. Computing the index ourselves keeps the @@ -171,13 +291,43 @@ Server's. 100k cap. - The time of the min or max value needs an arg-min style lookup. It is computed with a window function, or with a second statement if QuestDB lacks one; T0 decides. `AggregateEvaluator` then produces the values. + - **Confirmed forms (T0 Q8).** They mirror `SqlAggregateQueries`'s three families and column order. They were checked + against hand-computed seeds and a 60k-row randomized cross-check across 4 partitions: 617 buckets, 0 mismatches. + - Classified subquery `s`: + ```sql + SELECT (cast(ts AS LONG) * 10 - $1) / $2 AS b, ts, kind, num_value, int_value, str_value, + CASE WHEN status < 2147483648 AND kind <> 0 THEN 1 ELSE 0 END AS u, + CASE WHEN status < 1073741824 AND kind <> 0 THEN 1 ELSE 0 END AS g + FROM {p}sample WHERE tag = $3 AND ts >= $4 AND ts < $5 AND NOT deleted + ``` + - Counts (ordinals 1–5 after `b`): `count() AS total_count, sum(u) AS usable_count, sum(g) AS good_count, count() - + sum(u) AS bad_count, sum(CASE WHEN u = 1 AND kind IN (4, 5) THEN 1 ELSE 0 END) AS non_numeric_count`. + - **Simple:** `SELECT b, , avg(CASE WHEN u = 1 THEN num_value END) AS avg_value, min(CASE WHEN u = 1 THEN + num_value END) AS min_value, max(…) AS max_value, sum(…) AS sum_value FROM () GROUP BY b ORDER BY b`. + - **StartEnd:** + ```sql + WITH s AS (), st AS (SELECT b, FROM s GROUP BY b), + f AS (SELECT b, kind, num_value, int_value, str_value, ts, + row_number() OVER (PARTITION BY b ORDER BY ts) AS rf, + row_number() OVER (PARTITION BY b ORDER BY ts DESC) AS rl FROM s WHERE u = 1) + SELECT st.*, x.kind, x.num_value, x.int_value, x.str_value, x.ts, y.kind, y.num_value, y.int_value, y.str_value, y.ts + FROM st LEFT JOIN (SELECT * FROM f WHERE rf = 1) x ON x.b = st.b + LEFT JOIN (SELECT * FROM f WHERE rl = 1) y ON y.b = st.b ORDER BY st.b + ``` + - **ActualTime:** the same `s`/`st`, then `r AS (SELECT b, num_value, ts, row_number() OVER (PARTITION BY b ORDER + BY num_value ASC|DESC, ts ASC) AS rn FROM s WHERE u = 1 AND num_value IS NOT NULL) SELECT st.*, r.num_value, r.ts + FROM st LEFT JOIN (SELECT * FROM r WHERE rn = 1) r ON r.b = st.b ORDER BY st.b`. + - A bucket with no usable sample returns NULL times but **`kind = 0`** (BYTE has no NULL). The mapper keys absence on + the NULL `ts`. + - `first()/last()` and `arg_min/arg_max` also returned correct values, but no `ORDER BY` fixes their tie order. The + window forms state the earliest-timestamp tie-break explicitly and are the ones to use. - **Errors.** A failed read fails only that node, under the same data-source contract as SQL Server. ## 4. Configuration (`ServerHistorian:QuestDb`) | Key | Default | Validation | |---|---|---| -| `ConnectionString` | — | Required when `Backend=QuestDb` and a consumer is enabled. An Npgsql string, e.g. `Host=questdb;Port=8812;Username=admin;Password=quest;Database=qdb;ServerCompatibilityMode=NoTypeLoading` (T0 confirms the exact keywords). | +| `ConnectionString` | — | Required when `Backend=QuestDb` and a consumer is enabled. An Npgsql string. **T0-confirmed:** `Host=questdb;Port=8812;Username=admin;Password=quest;Database=qdb;ServerCompatibilityMode=NoTypeLoading` (the docker-dev overlay from the host: `Host=localhost;Port=18812;…`). `ServerCompatibilityMode=NoTypeLoading` is kept: it skips the `pg_type` load, and every confirmed form was run with it. The string also works without it. `No Reset On Close` is **not** needed, because pooled open → close → reopen works. | | `TablePrefix` | `hist_` | `^[A-Za-z0-9_]+$`. Only checked when a consumer is enabled. | | `RetentionDays` | unset | ≥ 1 when set | | `CommandTimeout` | `00:00:30` | > 0 | diff --git a/docs/plans/2026-10-10-questdb-spike.md b/docs/plans/2026-10-10-questdb-spike.md new file mode 100644 index 00000000..510dc1cc --- /dev/null +++ b/docs/plans/2026-10-10-questdb-spike.md @@ -0,0 +1,411 @@ +# QuestDB historian backend — T0 feasibility spike + +**Status:** run 2026-10-11 against a live container. Gitea #666. Spec: `docs/plans/2026-10-10-questdb-historian-design.md` +(§2–§4 amended in place from this doc; every confirmed form there is marked **T0-confirmed**). + +## Fixture + +| Item | Value | +|---|---| +| Image | **`questdb/questdb:10.0.1`** — Docker Hub `tag_last_pushed` **2026-08-24T14:00:08Z** (48 days before 2026-10-10) | +| Why that tag | Newest exact `x.y.z` tag ≥ 14 days older than 2026-10-10. Newer tags are `nightly*` / `patch-*` only. Floor for TTL is **8.2.3**; 10.0.1 is well above it. | +| Build | `SELECT build()` → `Build Information: QuestDB 10.0.1, JDK 25.0.2, Commit Hash 7a391566ac827e0d8c269b91f2415c16b0a32a84` | +| Overlay | `docker-dev/docker-compose.questdb.yml` — container `otopcua-dev-questdb-1`, ports `18812:8812` (PG wire) and `19000:9000` (console/REST), named volume `otopcua-questdb-data` (explicit `name:` so it is not project-prefixed) | +| Up | `docker compose -f docker-dev/docker-compose.yml -f docker-dev/docker-compose.questdb.yml up -d --no-deps questdb` (a `--dry-run` showed only the volume + `otopcua-dev-questdb-1` being created; nothing else in the `otopcua-dev` project is touched) | +| Healthcheck | the image ships `/usr/bin/curl` (`command -v curl` → `/usr/bin/curl`; no `wget`), so `curl -fs http://localhost:9003/status` is kept. Container reached `healthy`. | +| Harness | throwaway net10.0 console app, Npgsql **10.0.3** (the repo's central pin, `Directory.Packages.props` line 97), in the session scratchpad `questdb-spike/` with its own `Directory.Build.props`, `Directory.Packages.props` and `nuget.config`. Nothing was added to the repo. | + +### Connection string (confirmed) + +``` +Host=localhost;Port=18812;Username=admin;Password=quest;Database=qdb;ServerCompatibilityMode=NoTypeLoading +``` + +Inside the docker network the same string uses `Host=questdb;Port=8812`. + +## Q1 — keywords, parameter types, CLR types + +``` + [OK] open+SELECT 1 with 'Host=localhost;Port=18812;Username=admin;Password=quest;Database=qdb' + [OK] open+SELECT 1 with '...;ServerCompatibilityMode=NoTypeLoading' + [OK] open->close->reopen x5 on one data source (no 'No Reset On Close') + [OK] reopen after a parameterised INSERT (prepared-ish state) + [FAIL] ts Utc + Timestamp: ArgumentException: Cannot write DateTime with Kind=UTC to PostgreSQL type 'timestamp without time zone', consider using 'timestamp with time zone'. ... + [OK] ts Utc + TimestampTz + [OK] ts Utc + inferred + [OK] ts Unspecified + Timestamp + [FAIL] ts Local + TimestampTz: ArgumentException: Cannot write DateTime with Kind=Local to PostgreSQL type 'timestamp with time zone', only UTC is supported. ... + [OK] kind bound as Integer (int4) + [OK] tag bound as Text + [OK] NaN double insert + [OK] +Inf double insert + cols: ts:timestamp without time zone/DateTime, tag:character varying/String, kind:smallint/Int16, num_value:double precision/Double, + int_value:bigint/Int64, str_value:character varying/String, status:bigint/Int64, deleted:boolean/Boolean, server_ts:timestamp without time zone/DateTime + 2026-01-02T03:04:05.1234560(Unspecified) | ts Utc + TimestampTz | 3 | 1.5 | 7 | s | 2147483648 | False | 2026-10-11T04:56:27.1103710(Unspecified) + ... + 2026-01-02T03:04:11.1234560(Unspecified) | nan | 0 | NULL | NULL | NULL | NULL | False | NULL + 2026-01-02T03:04:12.1234560(Unspecified) | inf | 0 | NULL | NULL | NULL | NULL | False | NULL + integer arithmetic with $n: + cols: ts::LONG - $1:bigint/Int64 [OK] cast(ts AS LONG) - $1 (Bigint) + cols: (ts::LONG * 10 - $1) / $2:bigint/Int64 [OK] (Bigint) [OK] (Integer) +``` + +Sentinel NULLs (extra check): + +``` + cols: ts, l:bigint/Int64, i:integer/Int32, b:smallint/Int16, z:boolean/Boolean, l = NULL:boolean + 2026-03-01T00:00:00 | NULL | NULL | 0 | False | True <- l = long.MinValue, i = int.MinValue written + 2026-03-01T00:00:01 | -9223372036854775807 | -2147483647 | 0 | False | False + 2026-03-01T00:00:02 | NULL | NULL | 0 | False | True <- BYTE/BOOLEAN bound as DBNull +``` + +**Decisions** +- Both connection strings work. **Keep `ServerCompatibilityMode=NoTypeLoading`**: it skips the `pg_type` load, and every + form in this doc was run with it on. **`No Reset On Close` is not needed**: open → close → reopen on one pooled + `NpgsqlDataSource` works. +- **Timestamps are bound as `NpgsqlDbType.TimestampTz` with `DateTimeKind.Utc`.** `Timestamp` + `Utc` throws in Npgsql + before reaching the server. `Unspecified` + `Timestamp` also works, but one rule is enough. +- **Timestamps read back as `DateTime` with `Kind=Unspecified`**, wall clock = UTC. The reader must + `DateTime.SpecifyKind(r.GetDateTime(i), DateTimeKind.Utc)`. +- CLR types returned: `BYTE` → `smallint` / **`Int16`** (`GetInt16`); `SYMBOL` → `character varying` / `String`; + `VARCHAR` → `String`; `LONG` → `Int64`; `INT` → `Int32`; `BOOLEAN` → `Boolean`; `TIMESTAMP` → `DateTime`; + `DOUBLE` → `Double`. `count()`/`sum(int)` → `Int64`. +- Parameter types: `BYTE` ← `Smallint` (`Integer` also accepted); `SYMBOL`/`VARCHAR` ← `Varchar` (`Text` also accepted); + `LONG` ← `Bigint`; `BOOLEAN` ← `Boolean`; `DOUBLE` ← `Double`; a NULL is `DBNull.Value` with the column's + `NpgsqlDbType`. +- `$n` in integer arithmetic: **bind as `NpgsqlDbType.Bigint`** (Integer also worked for small values; Bigint is the + rule because tick offsets exceed int32). +- **DOUBLE NaN and ±Inf read back as NULL.** **`LONG` `long.MinValue` and `INT` `int.MinValue` are QuestDB's NULL + sentinels** and read back NULL. **`BYTE` and `BOOLEAN` have no NULL**: unset or DBNull reads `0` / `false`. So + `deleted` defaults to `false` and `kind` of a never-written field is `0` (Empty). +- **Every server error carries `SqlState = 00000`.** Classification cannot use SQLSTATE: a `PostgresException` is a + server-side rejection (decide by `MessageText`), any other `NpgsqlException`/`IOException`/`SocketException`/timeout + is transport. + +## Q2 — multi-row parameterised INSERT and batch size + +``` + [OK] 1000 rows x 9 params = 9000 params 88 ms + [OK] 2000 rows x 9 params = 18000 params 167 ms + [OK] 3640 rows x 9 params = 32760 params 342 ms + [FAIL] 3641 rows x 9 params = 32769 params: PostgresException: 00000: invalid parameter count [parameterCount=-32767, offset=225670 + [FAIL] 5000 rows x 9 params = 45000 params: PostgresException: 00000: invalid parameter count [parameterCount=-20536, ... + [FAIL] 7282 rows x 9 params = 65538 params: NpgsqlException: A statement cannot have more than 65535 parameters + 20 x 1000-row statements: fails=0, 122 ms total + count()=27251 | count_distinct(ts)=27251 expected rows = 27251 +``` + +**Decision:** QuestDB reads the Bind parameter count as a **signed int16**, so the real limit is **32,767 parameters per +statement**, not Npgsql's 65,535. **Batch size = 1,000 rows per `INSERT … VALUES (…),(…)`** (9,000 parameters for +`hist_sample`), which is reliable and far under the limit. **DeleteRaw page size = 1,000 rows** (see the DDL section). + +## Q3 — `DEDUP UPSERT KEYS` + +``` + before: 2026-03-01T00:00:00 | a | 1 | False 2026-03-01T00:00:00 | b | 2 | False + after re-insert of (t0,'a') as tombstone: + 2026-03-01T00:00:00 | a | NULL | True 2026-03-01T00:00:00 | b | 2 | False + NOT deleted: 2026-03-01T00:00:00 | b | 2 + same key twice in one statement (10 then 20): 2026-03-01T00:00:01 | c | 20 + hist_event with event_id VARCHAR as a dedup key: + [OK] CREATE hist_event-shape with DEDUP UPSERT KEYS(ts, event_id) on VARCHAR + [OK] insert e1 'first', e2, then e1 'second' + 2026-03-01T00:00:00 | e1 | src1 | 500 | second | NULL + 2026-03-01T00:00:00 | e2 | src1 | 500 | other | NULL +``` + +**Decision:** re-inserting a key replaces the whole row; a tombstone written at `(ts, tag)` hides the row from `NOT +deleted`. **Within one statement the LAST duplicate wins**, so the writer's "first value wins" rule must be applied +client-side before the INSERT (spec §2.2 step 1 already does this). **`event_id VARCHAR` works as a dedup key**, so the +S4 fallback (no dedup on `hist_event`) is **not needed**. + +## Q4 — `wal_tables()` and the visibility wait + +``` + cols: name:varchar, suspended:boolean, writerTxn:bigint, bufferedTxnSize:bigint, sequencerTxn:bigint, + errorTag:varchar, errorMessage:varchar, memoryPressure:integer + run 0: seqTxn before=0 after=1 visibleImmediately=False waitMs=13 visibleAfterWait=True + run 1: seqTxn before=1 after=2 visibleImmediately=False waitMs=0 visibleAfterWait=True + ... + run 16: seqTxn before=16 after=17 visibleImmediately=False waitMs=6 visibleAfterWait=True + lag ms: min=0 median=0 p95=6 max=13; visible-without-wait 17/20 +``` + +(An earlier run: visible without waiting 13/20, max 6 ms.) Every autocommitted INSERT is exactly one sequencer +transaction. + +**Decision (the form T3's `QuestDbWalVisibility` implements):** +1. After the INSERT(s) return, on the same or any connection: + `SELECT sequencerTxn FROM wal_tables() WHERE name = $1` (`$1` = the table name, `Varchar`) → `target`. +2. Poll `SELECT writerTxn, suspended FROM wal_tables() WHERE name = $1` every ~10 ms until `writerTxn >= target`, or + `suspended` is true, or `VisibilityTimeout` elapses. + +A concurrent writer can only make `target` larger, which lengthens the wait slightly and never shortens it. The wait is +needed: up to 7/20 inserts were not yet visible on the next query. The S11 key-poll fallback is **not needed**. + +## Q5 — TTL + +``` + [FAIL] CREATE ... PARTITION BY DAY WAL DEDUP UPSERT KEYS(ts, tag) TTL 7 DAYS (TTL last): unexpected token [TTL] + [OK] CREATE ... PARTITION BY DAY TTL 7 DAYS WAL DEDUP UPSERT KEYS(ts, tag) (TTL before WAL) + spike_q5 | DAY | True | True | ttlValue=1 | ttlUnit=WEEK + [FAIL] ALTER TABLE spike_q5 SET TTL $1 DAYS (parameter): invalid syntax, should be but was $1 + [FAIL] TTL 7 DAYS on PARTITION BY YEAR table: TTL value must be an integer multiple of the partition size (its time interval) +Q5b ALTER ... SET TTL visibility in tables() + SET TTL 30 DAYS : seqTxn 6->7; applied after 100 ms; tables() now ttlValue=30 ttlUnit=DAY + SET TTL 14 DAYS : seqTxn 7->8; applied after 94 ms; tables() now ttlValue=2 ttlUnit=WEEK + SET TTL 1 DAY : seqTxn 8->9; applied after 59 ms; tables() now ttlValue=1 ttlUnit=DAY + SET TTL 0 DAYS : seqTxn 9->10; applied after 25 ms; tables() now ttlValue=0 ttlUnit=HOUR + SET TTL 3 DAYS : seqTxn 10->11; applied after 60 ms; tables() now ttlValue=3 ttlUnit=DAY + after inserting now-20d and now into a TTL 7 DAYS table: + new | 2026-10-11T05:03:03 +Q5c TTL reference clock (future-dated row): + after inserting a now+30d row, then one more ordinary commit: + d-3 | 2026-10-08 now | 2026-10-11 now+1h | 2026-10-11 d+30 | 2026-11-10 +``` + +Image `conf/server.conf`: `#cairo.ttl.use.wall.clock=true` — "when true (default), TTL enforcement uses wall clock +time to prevent accidental data loss when future timestamps are inserted". + +**Decision:** +- Create syntax: **`… TIMESTAMP(ts) PARTITION BY DAY TTL n DAYS WAL DEDUP UPSERT KEYS(…)`** — TTL goes **between + `PARTITION BY` and `WAL`**. Alter: **`ALTER TABLE t SET TTL n DAYS`**; remove: **`ALTER TABLE t SET TTL 0 DAYS`**. + `n` cannot be a parameter; it is the validated integer `RetentionDays`, formatted with `InvariantCulture`. +- Metadata: `SELECT ttlValue, ttlUnit FROM tables() WHERE table_name = $1`. **QuestDB normalises the unit**: 7 days + shows `1 WEEK`, 14 days `2 WEEK`, 30 days `30 DAY`, none `0 HOUR`. The bootstrapper compares in hours + (`HOUR`=1, `DAY`=24, `WEEK`=168; any other unit counts as different) and issues `ALTER` only when they differ. +- `ALTER … SET TTL` is a WAL transaction: `tables()` shows the old value until it is applied (25–100 ms). Do not + re-read to verify without the Q4 wait. +- Partitions older than TTL are dropped on commit. With the default wall-clock mode a future-dated sample does **not** + expire current data. +- TTL cannot go on the `PARTITION BY YEAR` `hist_tag` table; it is never needed there. + +## Q6 — timestamp precision + +``` + raw sent 2026-03-01T12:00:00.1234567(Utc) -> read 2026-03-01T12:00:00.1234560 micros=1772366400123456 exact=False + raw9 sent 2026-03-01T12:00:01.1234569(Utc) -> read 2026-03-01T12:00:01.1234560 micros=1772366401123456 exact=False + pre sent 2026-03-01T12:01:00.1234560(Utc) -> read 2026-03-01T12:01:00.1234560 micros=1772366460123456 exact=True +``` + +**Decision:** Npgsql **truncates** sub-µs ticks (…1234567 → …123456; not rounded). That is truncation **toward zero +relative to 2000-01-01**, so before 2000 it rounds *up* (Q13). A pre-truncated (`ticks - ticks % 10`) value +round-trips exactly. This confirms R6: always `Round` first. + +## Q7 — concurrent `CREATE TABLE IF NOT EXISTS` + +``` +Q7 concurrent CREATE TABLE IF NOT EXISTS (4 connections x 20 rounds) + errors: none + plain CREATE on existing: SqlState=00000 MessageText='table already exists' +``` + +(An earlier run with 2 connections × 20 rounds also had no errors.) **Decision:** `IF NOT EXISTS` was race-free here. +The S11 catch rule stays as defence: catch a `PostgresException` whose `MessageText` contains **`already exists`**, +then re-check with `SELECT count() FROM tables() WHERE table_name = $1`, and rethrow when the table is not there. Never +key on SQLSTATE (it is always `00000`). + +## Q8 — bucket statistics + +Seed (3 × 10 s buckets from `start = t0 + 0.5 s`): bucket 0 holds Good 5, Uncertain 2, Bad 9, a usable String, Good 2 +(a min tie with the Uncertain 2), a tombstone of 100, and Good 7 at the last µs of the bucket. Bucket 1 holds Good 1 at +the edge, an Empty `BadNoCommunication` and an Empty with Good status. Bucket 2 holds Bad 3 only. Another tag sits in +range. + +``` + Simple family (GROUP BY alias): + cols: b, total_count, usable_count, good_count, bad_count, non_numeric_count, avg_value, min_value, max_value, sum_value + 0 | 6 | 5 | 4 | 1 | 1 | 4 | 2 | 7 | 16 + 1 | 3 | 1 | 1 | 2 | 0 | 1 | 1 | 1 | 1 + 2 | 1 | 0 | 0 | 1 | 0 | NULL | NULL | NULL | NULL + [OK] simple GROUP BY (no subquery) [OK] simple implicit group-by + StartEnd form A (first()/last()): + 0 | 6 | 5 | 4 | 1 | 1 | 1 | 5 | NULL | NULL | 00:00:00.5000000 | 1 | 7 | NULL | NULL | 00:00:10.4999990 + 1 | 3 | 1 | 1 | 2 | 0 | 1 | 1 | NULL | NULL | 00:00:10.5000000 | 1 | 1 | NULL | NULL | 00:00:10.5000000 + 2 | 1 | 0 | 0 | 1 | 0 | 0 | NULL | NULL | NULL | NULL | 0 | NULL | NULL | NULL | NULL + StartEnd form B (row_number window): identical rows [OK] filter-in-subquery [OK] extra predicate in ON + ActualTime (min): 0 ... | 2 | 00:00:01.5000000 1 ... | 1 | 00:00:10.5000000 2 ... | NULL | NULL + ActualTime (max): 0 ... | 7 | 00:00:10.4999990 1 ... | 1 | 00:00:10.5000000 2 ... | NULL | NULL + [OK] arg_min(ts, num_value) exists? 0 | 00:00:01.5 | 00:00:10.499999 1 | 00:00:10.5 | 00:00:10.5 + randomized cross-check (60k rows over 3 days (4 partitions), 2 tags, 7 min buckets, random statuses/kinds/tombstones): + buckets=617 rowsInWindow=28423 mismatches=0; first()/last() 23 ms, window 21 ms, actual-time min+max 26 ms; + arg_min/arg_max (one stmt) 11 ms, tie-break mismatches vs earliest-ts rule = 0 +``` + +Every hand-computed value matches: bucket 0 averages (5+2+2+7)/4 = 4, the tombstone is invisible, the sample one µs +before the edge stays in bucket 0, the edge sample is in bucket 1, and both Empty rows count as bad whatever their +status. The randomized cross-check compares counts, first/last usable times, sums and min/max actual times against a +client-side computation. It used a µs-aligned start that is not on a second boundary, and integer-rounded values to +force ties. + +**Decision (the confirmed forms, with the SQL Server family split kept):** +- Classified subquery `s`. Status is stored as the **unsigned** uint in a LONG, so usable = `status < 2147483648` and + good = `status < 1073741824`, both also needing `kind <> 0`: + ```sql + SELECT (cast(ts AS LONG) * 10 - $1) / $2 AS b, ts, kind, num_value, int_value, str_value, + CASE WHEN status < 2147483648 AND kind <> 0 THEN 1 ELSE 0 END AS u, + CASE WHEN status < 1073741824 AND kind <> 0 THEN 1 ELSE 0 END AS g + FROM {p}sample WHERE tag = $3 AND ts >= $4 AND ts < $5 AND NOT deleted + ``` + `$1` = window start in **ticks since the Unix epoch** (`start.Ticks - DateTime.UnixEpoch.Ticks`, `Bigint`); `$2` = + interval in ticks (`Bigint`); `$4`/`$5` = rounded window bounds. `cast(ts AS LONG)` is µs since 1970, so `*10` is + ticks. That is exact because both sides are already µs-aligned (R6). At year 9999 `µs*10` ≈ 2.5e18 < 9.2e18, so it + cannot overflow. +- Common counts: `count() AS total_count, sum(u) AS usable_count, sum(g) AS good_count, count() - sum(u) AS bad_count, + sum(CASE WHEN u = 1 AND kind IN (4, 5) THEN 1 ELSE 0 END) AS non_numeric_count`. +- **Simple:** `SELECT b, , avg(CASE WHEN u = 1 THEN num_value END) AS avg_value, min(…), max(…), sum(…) FROM () + GROUP BY b ORDER BY b`. +- **StartEnd:** the window form, which is the same shape as SQL Server: + `WITH s AS (), st AS (SELECT b, FROM s GROUP BY b), f AS (SELECT b, kind, num_value, int_value, + str_value, ts, row_number() OVER (PARTITION BY b ORDER BY ts) AS rf, row_number() OVER (PARTITION BY b ORDER BY ts + DESC) AS rl FROM s WHERE u = 1) SELECT st.*, x.kind, x.num_value, x.int_value, x.str_value, x.ts, y.kind, …, y.ts + FROM st LEFT JOIN (SELECT * FROM f WHERE rf = 1) x ON x.b = st.b LEFT JOIN (SELECT * FROM f WHERE rl = 1) y ON + y.b = st.b ORDER BY st.b`. A bucket without a usable sample has NULL `ts` but **kind `0`** (BYTE has no NULL), so the + evaluator must key "no first/last" on the NULL time. +- **ActualTime:** `… r AS (SELECT b, num_value, ts, row_number() OVER (PARTITION BY b ORDER BY num_value ASC|DESC, ts + ASC) AS rn FROM s WHERE u = 1 AND num_value IS NOT NULL) SELECT st.*, r.num_value, r.ts FROM st LEFT JOIN (SELECT * + FROM r WHERE rn = 1) r ON r.b = st.b ORDER BY st.b`. +- **One query per node holds for every family.** The S11 two-statement fallback is not needed. `first()/last()` and + `arg_min/arg_max` also gave correct answers, but their tie order is not specified by an `ORDER BY`. The window form + states the earliest-timestamp tie-break explicitly, so it is the one to use. +- `SAMPLE BY` is not used. + +## Q9 — AtTime neighbour lookup + +200k rows (2 tags × 100k, ~1.5 s apart, 5 % tombstones). 102 request times, including one before the first sample, +one exact hit and one after the last sample. + +``` + form A SQL (one time): SELECT * FROM (SELECT 0 AS req, 0 AS dir, ts, kind, num_value, int_value, str_value, status FROM spike_q9 + WHERE tag = $1 AND ts <= $2 AND NOT deleted ORDER BY ts DESC LIMIT 1) UNION ALL SELECT * FROM (SELECT 0 AS req, 1 AS dir, ts, ... + WHERE tag = $1 AND ts > $2 AND NOT deleted ORDER BY ts ASC LIMIT 1) + form A chunk= 1: 61 ms for 102 times, rows=202, mismatches=0 + form A chunk= 10: 20 ms ... chunk= 25: 18 ms chunk= 50: 19 ms chunk= 60: 18 ms chunk= 63: 16 ms (all mismatches=0) + form A chunk= 64: FAIL 'SQL model is too complex to evaluate' (128 UNION ALL branches) + form A chunk=102: FAIL 'SQL model is too complex to evaluate' (204 UNION ALL branches) + form B: 1 ms for 102 times (prev only), rows=102, mismatches=0 [ASOF JOIN of a long_sequence(1) UNION ALL request set] +``` + +**Decision: form A, chunked `LIMIT 1` seeks, at most 50 request times (100 `UNION ALL` branches) per statement.** +QuestDB rejects 128 or more branches. The literal `req` index and `dir` (0 = previous-or-equal, 1 = next) columns are +generated integers, not user input; `$1` is the tag and `$2..` the rounded times. It returns both neighbours in one +shape and costs ~0.2 ms per time. ASOF JOIN is faster, but it only gives the previous neighbour, so a second +seek form would still be needed for the next one. That does not justify two code paths at this scale. + +## Q10 — long strings and SYMBOL capacity + +``` + [OK] 400-char tag (SYMBOL) + 400-char str_value (VARCHAR) + [OK] 5000-char str_value + 30,000 distinct SYMBOL values inserted in 723 ms + count()=30002 | count_distinct(tag)=30002 + 400-char round trip: tag equal=True str equal=True lengths=400,400 (includes non-ASCII αβγ…) + tag | SYMBOL | symbolCached=True | symbolCapacity=16384 + point lookup by tag among 30k symbols: count=1 0 ms +``` + +**Decision:** VARCHAR and SYMBOL take 400 characters (and VARCHAR takes 5,000). SYMBOL grows past its capacity hint +(16,384) without error, so 30k tags is fine. No `CAPACITY` clause is needed. + +## Q11 — existing-timestamp lookup for a batch + +``` + [OK] range form: WHERE tag=$1 AND ts >= $2 AND ts <= $3 (tombstones included) -> 10 rows, incl. 00:00:08 deleted=True + [OK] IN-list form: ts IN ($2..$11) -> 4 rows (00, 06, 12, 18) + [OK] tombstone visible to range lookup -> 00:00:08 | True +``` + +**Decision:** both work with parameters. **Live writer (§2.2 step 2): the range form** +`SELECT ts FROM {p}sample WHERE tag = $1 AND ts >= $2 AND ts <= $3` (no `NOT deleted`, so tombstones count as present), +then a client-side set test. **Updater (sparse item times): the IN-list form** `SELECT ts, deleted FROM {p}sample WHERE +tag = $1 AND ts IN ($2, …)`, chunked at 1,000 times. + +## Q12 — test cleanup and prefix listing + +``` + LIKE 't12_%' (note: _ is a wildcard) -> t12_sample, t12xsample <- WRONG: matches t12xsample + LIKE $1 with escaped _ ('t12\_%') -> t12_sample + starts_with(table_name, $1) -> t12_sample + left(table_name, 4) = $1 -> t12_sample + [OK] DROP TABLE IF EXISTS t12_sample (exists) [OK] DROP TABLE IF EXISTS t12_sample (already gone) +``` + +**Decision:** cleanup is `DROP TABLE IF EXISTS {name}`. List with **`SELECT table_name FROM tables() WHERE +starts_with(table_name, $1)`**. Never use `LIKE 'prefix%'`, because `_` in `hist_` is a wildcard. + +## Q13 — binding extreme DateTimes + +Seeds: 1970-01-01, 1990-05-05T00:00:00.123456, 2026-03-01, 2200-01-01. "Round" is R6's rule: `MinValue` → `+10 ticks`, +then `ticks - ticks % 10`. + +``` + DateTime.MinValue raw -> server sees µs -9222425352054775808 (Npgsql sends -infinity) + DateTime.MinValue Round -> 0001-01-01T00:00:00.0000010 (µs -62135596799999999) + DateTime.MaxValue raw -> server sees µs -9222425352054775809 (Npgsql sends +infinity) + DateTime.MaxValue Round -> 9999-12-31T23:59:59.9999990 (µs 253402300799999999) + 0001-01-01 raw -> µs -9222425352054775808 (== MinValue -> -infinity) + 1601-01-01 raw/Round -> 1601-01-01T00:00:00 (µs -11644473600000000) + 9999-12-31T23:59:59.9999999 raw -> µs -9222425352054775809 (== MaxValue -> +infinity) + 1969-12-31T23:59:59.9999995 raw -> 1970-01-01T00:00:00.0000000 (µs 0) Round -> 1969-12-31T23:59:59.9999990 (µs -1) + 1990-05-05T00:00:00.1234567 raw -> 1990-05-05T00:00:00.1234570 Round -> ...1234560 + 1990-05-05T00:00:00.1234565 raw -> 1990-05-05T00:00:00.1234570 Round -> ...1234560 + 2026-03-01T00:00:00.0000005 raw -> 2026-03-01T00:00:00.0000000 Round -> same + range queries WHERE ts >= $1 AND ts < $2: + Min..Max raw count=0 <- silently empty + Min..Max Round count=4 + 0001..9999.9999999 Round count=4 + 1601..t0 Round count=2 + 1990.1234565..1990.1234575 raw count=0 <- seed .123456 missed (bound arrived as .123457) + 1990.1234565..1990.1234575 Round count=1 + inserting extreme designated timestamps: + [FAIL] insert ts=1969-12-31T23:59:59: PostgresException: 00000: designated timestamp before 1970-01-01 is not allowed + [FAIL] insert ts=1601-01-01: ... [FAIL] insert ts=Min+10 ticks: ... + [OK] insert ts=9999-12-31 Round -> 9999-12-31T23:59:59.9999990 +``` + +**Decision:** R6's `BindTimestamp` is **confirmed necessary and sufficient**. +- A raw `DateTime.MinValue`/`MaxValue` (and anything equal to them) becomes ±infinity, and QuestDB stores that as a + near-`long.MinValue` value. A `[Min, Max)` range then matches **nothing**, with no error. +- A raw pre-2000 sub-µs value rounds **up**, which shifts a bound across a stored sample. +- With Round, every bound behaves. +- **New constraint:** a designated timestamp **before 1970-01-01** cannot be inserted. Reads with pre-1970 bounds work. + See the spec §2.2 ruling candidate. + +## DDL and DeleteRaw paging (extra) + +``` + [OK] CREATE TABLE IF NOT EXISTS spike_hist_sample (...) TIMESTAMP(ts) PARTITION BY DAY TTL 30 DAYS WAL DEDUP UPSERT KEYS(ts, tag) + [OK] CREATE TABLE IF NOT EXISTS spike_hist_event (...) TIMESTAMP(ts) PARTITION BY DAY TTL 30 DAYS WAL DEDUP UPSERT KEYS(ts, event_id) + [OK] CREATE TABLE IF NOT EXISTS spike_hist_tag (...) TIMESTAMP(ts) PARTITION BY YEAR WAL DEDUP UPSERT KEYS(ts, name) + [OK] re-run (idempotent) x3 + spike_hist_event | DAY | True | True | 30 | DAY + spike_hist_sample | DAY | True | True | 30 | DAY + spike_hist_tag | YEAR | True | True | 0 | HOUR + hist_tag upsert-by-name at ts=1970-01-01 (tagA twice, tagB once): + 1970-01-01T00:00:00 | tagA | 4 | degF | second + 1970-01-01T00:00:00 | tagB | 1 | NULL | NULL + DeleteRaw paging (page = 1000 rows): 2500 live rows + 1 outside range + pages=3 tombstones=2500 in 103 ms + deleted=False 1 deleted=True 2500 +``` + +DeleteRaw loop, as confirmed: `SELECT ts FROM {p}sample WHERE tag = $1 AND ts >= $2 AND ts < $3 AND NOT deleted ORDER BY +ts LIMIT 1000` → one 1,000-row tombstone INSERT → the Q4 wait → continue from `last ts + 10 ticks`. The wait matters: +without it the next page can re-read rows it just tombstoned. + +## Rulings candidates raised by the spike + +1. **Pre-1970 timestamps cannot be stored** (Q13). QuestDB rejects a designated timestamp before 1970-01-01, and the + whole multi-row INSERT fails. Proposed: + - the live writer drops such samples and logs one Warning per batch. Retrying cannot succeed, and sending the + sample would fail the tag's whole group into #538 quarantine; + - HistoryUpdate Insert/Replace/Update items before 1970 return `BadOutOfRange`; + - DeleteAtTime/DeleteRaw find nothing there and return `BadNoEntryExists`; + - reads are unaffected. + Conformance tests that seed pre-1970 data are skipped for QuestDB. +2. **`long.MinValue` is QuestDB's LONG NULL** (sentinel check). An Int64 sample of exactly `long.MinValue` would read + back `int_value` NULL, and `FromColumns` would then serve **0**. Proposed: the QuestDB read path repairs it. When + `kind = Int64`, `int_value IS NULL` and `num_value = -9.223372036854775808E18` (exact in a double), serve + `long.MinValue`. That is a local fix-up before the shared `FromColumns`, with no write-side change. +3. **TTL metadata is unit-normalised and applied asynchronously** (Q5). The bootstrapper compares in hours and does not + re-read straight after an `ALTER`. From 27991434a07fcf20028f4af08c329cfbbc69fc3f Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Sun, 11 Oct 2026 01:10:31 -0400 Subject: [PATCH 05/23] spike(historian): T0 review fixes Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe --- .../2026-10-10-questdb-historian-design.md | 31 ++++++++++---- docs/plans/2026-10-10-questdb-spike.md | 42 ++++++++++++++++++- 2 files changed, 63 insertions(+), 10 deletions(-) diff --git a/docs/plans/2026-10-10-questdb-historian-design.md b/docs/plans/2026-10-10-questdb-historian-design.md index 63ec36ed..c2f1ec99 100644 --- a/docs/plans/2026-10-10-questdb-historian-design.md +++ b/docs/plans/2026-10-10-questdb-historian-design.md @@ -262,9 +262,17 @@ Every query filters `NOT deleted`. Every value goes through `StoredSample`, so t Server's. - **Raw.** A range on `ts`, `ORDER BY ts [DESC]`, `LIMIT @top` (`maxValuesPerNode == 0` ⇒ unbounded), at most `top` samples, `ContinuationPoint: null`; the node manager pages by re-reading from the last timestamp (plan R8). The dedup key forbids same-instant ties. Bounds follow the existing data-source contract. + - **T0-confirmed form (review fix):** `SELECT ts, kind, num_value, int_value, str_value, status FROM {p}sample WHERE + tag = $1 AND ts >= $2 AND ts < $3 AND NOT deleted ORDER BY ts [DESC] LIMIT $4`. + - `$4` is a **bound** parameter, `NpgsqlDbType.Integer` (Int32) or `Bigint` (Int64); both work on 10.0.1, ASC and + DESC. Use `Integer`, since `top` is an `int`. + - `int.MaxValue` (the R8 "unbounded" value) and `long.MaxValue` both return every row. `LIMIT 0` returns no rows, + but R8 never sends 0. + - The inlined-literal fallback is not needed. - **AtTime (T0-confirmed Q9).** Find the previous-or-equal neighbour and the next neighbour of each requested time - with `LIMIT 1` seeks, `UNION ALL`-ed, **at most 50 request times (100 branches) per statement**. QuestDB rejects - 128 or more branches with `SQL model is too complex to evaluate`. Per time `k`, with `$1` = tag and `$j` = the + with `LIMIT 1` seeks, `UNION ALL`-ed, **at most 50 request times (100 branches) per statement**. The exact QuestDB + 10.0.1 limit is **126 `UNION ALL` branches per statement (63 request times)**: 126 work, and 127 fail with `SQL + model is too complex to evaluate`. So 50 times leaves a 26-branch margin. Per time `k`, with `$1` = tag and `$j` = the rounded time: ```sql SELECT * FROM (SELECT k AS req, 0 AS dir, ts, kind, num_value, int_value, str_value, status FROM {p}sample @@ -289,8 +297,9 @@ Server's. - It does **not** use `SAMPLE BY`, whose alignment rules differ from ours. Computing the index ourselves keeps the bucket edges identical to SQL Server: start inclusive, end exclusive, a partial final bucket, interval 0, and the 100k cap. - - The time of the min or max value needs an arg-min style lookup. It is computed with a window function, or with a - second statement if QuestDB lacks one; T0 decides. `AggregateEvaluator` then produces the values. + - The time of the min or max value (ActualTime family) comes from a `row_number()` window function in the same single + statement: `ORDER BY num_value ASC|DESC, ts ASC`, with ties going to the earliest sample (T0-confirmed; the form is + below). No second statement is used. `AggregateEvaluator` then produces the values. - **Confirmed forms (T0 Q8).** They mirror `SqlAggregateQueries`'s three families and column order. They were checked against hand-computed seeds and a 60k-row randomized cross-check across 4 partitions: 617 buckets, 0 mismatches. - Classified subquery `s`: @@ -362,11 +371,15 @@ The Gateway keys (`Endpoint`, `ApiKey`, `UseTls`) are not required for QuestDb, ## 6. Environment -- docker-dev gains a `questdb` service, using an exact image tag that T0 pins (at least two weeks old). It sits behind - a compose profile (`questdb`) so the default rig is unchanged. -- It exposes PG wire `8812` and the web console `9000` on host ports that T0 chooses so they don't collide with - AdminUI `:9000`/`:9200`. -- It uses a named volume. +- QuestDB is an **optional overlay file**, `docker-dev/docker-compose.questdb.yml`, so the default rig is unchanged. + It defines the `questdb` service: + - image `questdb/questdb:10.0.1`, pinned by T0 and pushed 2026-08-24; + - container `otopcua-dev-questdb-1`; + - started with `docker compose -f docker-dev/docker-compose.yml -f docker-dev/docker-compose.questdb.yml up -d + --no-deps questdb`. +- It exposes PG wire on host port **18812** (→ 8812) and the web console/REST on host port **19000** (→ 9000), so it + does not collide with AdminUI `:9000`/`:9200`. +- It uses the named volume `otopcua-questdb-data`. - The central nodes gain `ServerHistorian__QuestDb__ConnectionString` (`OTOPCUA_HISTORIAN_QUESTDB`), defaulting to the `questdb` service. - No new NuGet packages. diff --git a/docs/plans/2026-10-10-questdb-spike.md b/docs/plans/2026-10-10-questdb-spike.md index 510dc1cc..1a028cab 100644 --- a/docs/plans/2026-10-10-questdb-spike.md +++ b/docs/plans/2026-10-10-questdb-spike.md @@ -287,7 +287,9 @@ one exact hit and one after the last sample. ``` **Decision: form A, chunked `LIMIT 1` seeks, at most 50 request times (100 `UNION ALL` branches) per statement.** -QuestDB rejects 128 or more branches. The literal `req` index and `dir` (0 = previous-or-equal, 1 = next) columns are +The exact limit is **126 `UNION ALL` branches (63 request times) per statement**. A re-measure with single-seek +branches (review fix, below) showed 126 work and 127 fail. The form A runs above agree: 63 times (126 branches) work, +and 64 times (128 branches) fail. The literal `req` index and `dir` (0 = previous-or-equal, 1 = next) columns are generated integers, not user input; `$1` is the tag and `$2..` the rounded times. It returns both neighbours in one shape and costs ~0.2 ms per time. ASOF JOIN is faster, but it only gives the previous neighbour, so a second seek form would still be needed for the next one. That does not justify two code paths at this scale. @@ -393,6 +395,44 @@ DeleteRaw loop, as confirmed: `SELECT ts FROM {p}sample WHERE tag = $1 AND ts >= ts LIMIT 1000` → one 1,000-row tombstone INSERT → the Q4 wait → continue from `last ts + 10 ticks`. The wait matters: without it the next page can re-read rows it just tombstoned. +## Review fixes (2026-10-11) + +### Raw `LIMIT $n` with a bound parameter + +`dotnet run -- limit` against 10.0.1, with 10 rows seeded in `[t0, t0+10 s)`. The SQL was `SELECT ts, num_value FROM +spike_lim WHERE tag = $1 AND ts >= $2 AND ts < $3 AND NOT deleted ORDER BY ts [DESC] LIMIT $4`: + +``` + rows=3 first=0 last=2 [OK] ASC LIMIT $4 (Integer 3) + rows=3 first=9 last=7 [OK] DESC LIMIT $4 (Integer 3) + rows=3 first=0 last=2 [OK] ASC LIMIT $4 (Bigint 3) + rows=3 first=9 last=7 [OK] DESC LIMIT $4 (Bigint 3) + rows=10 first=0 last=9 [OK] ASC LIMIT $4 (Integer int.MaxValue) + rows=10 first=9 last=0 [OK] DESC LIMIT $4 (Integer int.MaxValue) + rows=10 first=0 last=9 [OK] ASC LIMIT $4 (Bigint long.MaxValue) + rows=10 first=9 last=0 [OK] DESC LIMIT $4 (Bigint long.MaxValue) + rows=0 first=NULL last=NULL [OK] ASC LIMIT $4 (Integer 0) + rows=0 first=NULL last=NULL [OK] DESC LIMIT $4 (Integer 0) +``` + +**Decision:** a bound `LIMIT $n` works, with `NpgsqlDbType.Integer` (Int32) or `Bigint` (Int64). Use `Integer`, since +`top` is an `int`. R8's unbounded value `int.MaxValue` returns every row. `LIMIT 0` returns nothing, but R8 never sends +it. The inlined-literal fallback is not needed. + +### Exact `UNION ALL` branch limit + +`dotnet run -- union`. Each branch is `SELECT * FROM (SELECT k AS req, ts FROM spike_un WHERE tag = $1 AND ts <= $2 AND +NOT deleted ORDER BY ts DESC LIMIT 1)`: + +``` + [OK] 100 branches [OK] 120 branches [OK] 125 branches [OK] 126 branches + [FAIL] 127 branches: PostgresException: 00000: SQL model is too complex to evaluate + [FAIL] 128 branches / 129 / 130: same +``` + +**Decision:** the limit is **126 branches, which is 63 AtTime request times** (two branches per time). The chosen chunk +of **50 times (100 branches)** is unchanged. + ## Rulings candidates raised by the spike 1. **Pre-1970 timestamps cannot be stored** (Q13). QuestDB rejects a designated timestamp before 1970-01-01, and the From 8c6d6aff6a352045e45cdc5cdf5c761569921aad Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Sun, 11 Oct 2026 01:26:49 -0400 Subject: [PATCH 06/23] refactor(historian): extract Driver.Historian.Sql.Common from the SQL Server backend (no behaviour change) (1a) Moves the database-independent logic of the SQL Server historian backend into a new shared library for the upcoming QuestDB backend (#666): status codes and bits, value/column mapping, StoredSample + quality clamp, the AtTime and aggregate evaluators, BucketPlan, LiveWritePreparer and HistoryUpdatePlanner (both with a resolutionTicks parameter, default 1), tag-name rules, the bootstrap failure cache, health counters, alarm event ids, the event continuation and severity clamp. The SQL Server backend now calls it; its T-SQL, TVP builders, connection, bootstrapper and catalog stay. Tests: StatusCodeOracleTests, ValueWriterPrepareTests, the column half of SampleRowMapperTests and ProcessedReadTests.Every_enum_member_is_either_served_or_refused move to the new Sql.Common.Tests under the same names. SqlServer integration tests get mechanical renames only. Test-name set across the three projects is identical to the baseline (160 tests). Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe --- ZB.MOM.WW.OtOpcUa.slnx | 2 + .../AggregateEvaluator.cs | 78 ++++++ .../AggregateFamily.cs | 33 +++ .../AlarmEventIds.cs | 61 +++++ .../AtTimeEvaluator.cs | 67 +++++ .../BootstrapFailureCache.cs | 46 ++++ .../BucketPlan.cs | 91 +++++++ .../BucketRow.cs | 57 +++++ .../EventContinuation.cs | 11 + .../HistorianDataTypes.cs} | 8 +- .../HistorianHealthCounters.cs | 70 ++++++ .../HistorianInfoBits.cs | 32 +++ .../HistorianSampleRow.cs | 9 + .../HistorianSchemaMissingException.cs | 9 + .../HistorianStatusCodes.cs | 27 ++ .../HistorianTagNames.cs | 17 ++ .../HistoricalEventRows.cs | 10 + .../HistoryUpdatePlanner.cs | 175 +++++++++++++ .../LiveWritePreparer.cs | 54 ++++ .../SampleColumns.cs | 66 +++++ .../StatusBits.cs | 33 +++ .../StoredSample.cs | 32 +++ ...OtOpcUa.Driver.Historian.Sql.Common.csproj | 22 ++ .../SampleRowMapper.cs | 61 +---- .../SqlAggregateQueries.cs | 31 +-- .../SqlHistorianStatusCodes.cs | 81 ------ .../SqlServerAlarmHistorianWriter.cs | 55 +---- .../SqlServerHistorianDataSource.cs | 230 +++--------------- .../SqlServerHistorianDataUpdater.cs | 115 +++------ .../SqlServerHistorianOptions.cs | 8 - .../SqlServerHistorianValueWriter.cs | 42 +--- .../SqlServerSchemaBootstrapper.cs | 22 +- .../SqlServerTagCatalog.cs | 14 +- ....OtOpcUa.Driver.Historian.SqlServer.csproj | 2 + ....WW.OtOpcUa.Core.Abstractions.Tests.csproj | 2 + .../ProcessedReadTests.cs | 16 ++ .../SampleRowMapperTests.cs | 45 ++++ .../StatusCodeOracleTests.cs | 10 +- .../ValueWriterPrepareTests.cs | 18 +- ...a.Driver.Historian.Sql.Common.Tests.csproj | 28 +++ .../LongNameTests.cs | 1 + .../ProcessedReadTests.cs | 7 - .../QualityContractTests.cs | 5 +- .../RawReadTests.cs | 7 +- .../SchemaBootstrapIntegrationTests.cs | 9 +- .../UpdateTests.cs | 47 ++-- .../WriteTests.cs | 9 +- .../AlarmEventIdTests.cs | 19 +- .../SampleRowMapperTests.cs | 40 +-- 49 files changed, 1262 insertions(+), 672 deletions(-) create mode 100644 src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/AggregateEvaluator.cs create mode 100644 src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/AggregateFamily.cs create mode 100644 src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/AlarmEventIds.cs create mode 100644 src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/AtTimeEvaluator.cs create mode 100644 src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/BootstrapFailureCache.cs create mode 100644 src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/BucketPlan.cs create mode 100644 src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/BucketRow.cs create mode 100644 src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/EventContinuation.cs rename src/Drivers/{ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlHistorianDataTypes.cs => ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/HistorianDataTypes.cs} (72%) create mode 100644 src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/HistorianHealthCounters.cs create mode 100644 src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/HistorianInfoBits.cs create mode 100644 src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/HistorianSampleRow.cs create mode 100644 src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/HistorianSchemaMissingException.cs create mode 100644 src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/HistorianStatusCodes.cs create mode 100644 src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/HistorianTagNames.cs create mode 100644 src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/HistoricalEventRows.cs create mode 100644 src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/HistoryUpdatePlanner.cs create mode 100644 src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/LiveWritePreparer.cs create mode 100644 src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/SampleColumns.cs create mode 100644 src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/StatusBits.cs create mode 100644 src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/StoredSample.cs create mode 100644 src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.csproj delete mode 100644 src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlHistorianStatusCodes.cs create mode 100644 tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/ProcessedReadTests.cs create mode 100644 tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/SampleRowMapperTests.cs rename tests/Drivers/{ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.Tests => ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests}/StatusCodeOracleTests.cs (77%) rename tests/Drivers/{ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.Tests => ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests}/ValueWriterPrepareTests.cs (68%) create mode 100644 tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests.csproj diff --git a/ZB.MOM.WW.OtOpcUa.slnx b/ZB.MOM.WW.OtOpcUa.slnx index 959ac9ac..16290f6c 100644 --- a/ZB.MOM.WW.OtOpcUa.slnx +++ b/ZB.MOM.WW.OtOpcUa.slnx @@ -27,6 +27,7 @@ + @@ -101,6 +102,7 @@ + diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/AggregateEvaluator.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/AggregateEvaluator.cs new file mode 100644 index 00000000..f79c3747 --- /dev/null +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/AggregateEvaluator.cs @@ -0,0 +1,78 @@ +using ZB.MOM.WW.OtOpcUa.Core.Abstractions; + +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common; + +/// +/// Maps one bucket's statistics () to the OPC UA value and status of an aggregate. +/// +/// +/// +/// Per interval: no samples ⇒ BadNoData. PercentGood / PercentBad ⇒ 100 × good|bad / all over +/// every sample, status Good. Otherwise no usable sample ⇒ BadNoData; a numeric aggregate with a usable +/// String/DateTime sample ⇒ BadAggregateInvalidInputs; else the value, Good when every sample is Good, +/// UncertainDataSubNormal when any is Bad or Uncertain. Every status — including BadNoData — +/// carries the Calculated info bits. Timestamps are the bucket start, except Minimum/MaximumActualTime +/// (the sample's time; ties ⇒ earliest). +/// +/// +/// Values: Count is ; Start/End keep the sample's type; every other aggregate is +/// computed over the numeric value (Boolean = 0/1). These are the simple per-sample +/// definitions, not Part 13's interval-boundary (interpolated bound / time-weighted) semantics. +/// +/// +public static class AggregateEvaluator +{ + /// Evaluates over one non-empty bucket. + /// Requested aggregate. + /// The bucket's statistics. + /// The bucket's start time (the result timestamp). + /// Server timestamp. + /// The calculated value. + /// The aggregate is not one of the thirteen served. + public static DataValueSnapshot Evaluate(HistoryAggregateType aggregate, BucketRow r, DateTime bucketStart, DateTime now) + { + DataValueSnapshot Calc(object? value, uint status, DateTime? t = null) => + new(value, HistorianInfoBits.WithCalculated(status), t ?? bucketStart, now); + + if (r.Total == 0) return NoData(bucketStart, now); + switch (aggregate) + { + case HistoryAggregateType.PercentGood: return Calc(100d * r.Good / r.Total, HistorianStatusCodes.Good); + case HistoryAggregateType.PercentBad: return Calc(100d * r.Bad / r.Total, HistorianStatusCodes.Good); + } + if (r.Usable == 0) return NoData(bucketStart, now); + + var anyText = aggregate is HistoryAggregateType.Count or HistoryAggregateType.Start or HistoryAggregateType.End; + if (!anyText && r.NonNumeric > 0) + return Calc(null, HistorianStatusCodes.BadAggregateInvalidInputs); + + var status = r.Good == r.Total ? HistorianStatusCodes.Good : HistorianStatusCodes.UncertainDataSubNormal; + switch (aggregate) + { + case HistoryAggregateType.Count: return Calc(r.Usable, status); + case HistoryAggregateType.Average: return Calc(r.Avg, status); + case HistoryAggregateType.Minimum: return Calc(r.Min, status); + case HistoryAggregateType.Maximum: return Calc(r.Max, status); + case HistoryAggregateType.Total: return Calc(r.Sum, status); + case HistoryAggregateType.Range: return Calc(r.Max - r.Min, status); + case HistoryAggregateType.Start: return Calc(r.First.ToObject(), status); + case HistoryAggregateType.End: return Calc(r.Last.ToObject(), status); + case HistoryAggregateType.Delta: + return r.First.TryGetDouble(out var a) && r.Last.TryGetDouble(out var b) + ? Calc(b - a, status) + : Calc(null, HistorianStatusCodes.BadAggregateInvalidInputs); + case HistoryAggregateType.MinimumActualTime: + case HistoryAggregateType.MaximumActualTime: + return Calc(r.Extreme, status, r.ExtremeTime); + default: + throw new HistoryAggregateNotSupportedException(aggregate); + } + } + + /// The result for a bucket with no samples (or no usable sample): BadNoData + Calculated. + /// The bucket's start time. + /// Server timestamp. + /// The no-data value. + public static DataValueSnapshot NoData(DateTime t, DateTime now) => + new(null, HistorianInfoBits.WithCalculated(HistorianStatusCodes.BadNoData), t, now); +} diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/AggregateFamily.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/AggregateFamily.cs new file mode 100644 index 00000000..761a0cab --- /dev/null +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/AggregateFamily.cs @@ -0,0 +1,33 @@ +using ZB.MOM.WW.OtOpcUa.Core.Abstractions; + +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common; + +/// The query shape that serves an aggregate. One query per family. +public enum AggregateFamily +{ + /// Average / Minimum / Maximum / Total / Count / Range / PercentGood / PercentBad: one GROUP BY. + Simple, + + /// Start / End / Delta: first + last usable sample per bucket. + StartEnd, + + /// MinimumActualTime / MaximumActualTime: extreme usable sample per bucket, ties ⇒ earliest. + ActualTime, +} + +/// Which aggregates the SQL historian backends serve, and by which family. +public static class AggregateFamilies +{ + /// The family that serves , or null when the backends do not. + /// Requested aggregate. + /// The family, or null for an unsupported aggregate. + public static AggregateFamily? FamilyOf(HistoryAggregateType aggregate) => aggregate switch + { + HistoryAggregateType.Average or HistoryAggregateType.Minimum or HistoryAggregateType.Maximum + or HistoryAggregateType.Total or HistoryAggregateType.Count or HistoryAggregateType.Range + or HistoryAggregateType.PercentGood or HistoryAggregateType.PercentBad => AggregateFamily.Simple, + HistoryAggregateType.Start or HistoryAggregateType.End or HistoryAggregateType.Delta => AggregateFamily.StartEnd, + HistoryAggregateType.MinimumActualTime or HistoryAggregateType.MaximumActualTime => AggregateFamily.ActualTime, + _ => null, + }; +} diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/AlarmEventIds.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/AlarmEventIds.cs new file mode 100644 index 00000000..5dbaf08b --- /dev/null +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/AlarmEventIds.cs @@ -0,0 +1,61 @@ +using System.Security.Cryptography; +using System.Text; +using ZB.MOM.WW.OtOpcUa.Core.Abstractions; +using ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian; + +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common; + +/// +/// The deterministic alarm-event id and numeric severity the SQL historian backends store, so the store-and-forward +/// drain's at-least-once redelivery (and both pair nodes writing the same event) converge on one row. +/// +public static class AlarmEventIds +{ + /// + /// Lowercase-hex SHA-256 over length-prefixed UTF-8 fields (little-endian int32 byte length, -1 for null): + /// EquipmentPath, AlarmId, EventKind, TimestampUtc.Ticks (decimal), User, Comment, Message. Length-prefixing keeps + /// field boundaries unambiguous ("a|b"+"c" ≠ "a"+"b|c"). + /// + /// The event. + /// 64 hex characters. + public static string ComputeEventId(AlarmHistorianEvent e) + { + using var ms = new MemoryStream(); + Field(ms, e.EquipmentPath); + Field(ms, e.AlarmId); + Field(ms, e.EventKind); + Field(ms, e.TimestampUtc.Ticks.ToString(System.Globalization.CultureInfo.InvariantCulture)); + Field(ms, e.User); + Field(ms, e.Comment); + Field(ms, e.Message); + return Convert.ToHexStringLower(SHA256.HashData(ms.ToArray())); + } + + /// The numeric Part 9 priority stored for a severity (same values the gateway path sends). + /// Alarm severity. + /// 200 / 500 / 700 / 900; 500 for anything else. + public static int ToNumericSeverity(AlarmSeverity s) => s switch + { + AlarmSeverity.Low => 200, + AlarmSeverity.Medium => 500, + AlarmSeverity.High => 700, + AlarmSeverity.Critical => 900, + _ => 500, + }; + + private static void Field(MemoryStream ms, string? value) + { + Span len = stackalloc byte[4]; + if (value is null) + { + System.Buffers.Binary.BinaryPrimitives.WriteInt32LittleEndian(len, -1); + ms.Write(len); + return; + } + + var bytes = Encoding.UTF8.GetBytes(value); + System.Buffers.Binary.BinaryPrimitives.WriteInt32LittleEndian(len, bytes.Length); + ms.Write(len); + ms.Write(bytes); + } +} diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/AtTimeEvaluator.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/AtTimeEvaluator.cs new file mode 100644 index 00000000..9440b843 --- /dev/null +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/AtTimeEvaluator.cs @@ -0,0 +1,67 @@ +using ZB.MOM.WW.OtOpcUa.Core.Abstractions; + +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common; + +/// The AtTime rules, applied to one requested time and its stored neighbours. +public static class AtTimeEvaluator +{ + /// + /// Applies the AtTime rules to one requested time (the result is stamped t): + /// + /// prev at exactly t ⇒ the stored value and status, unchanged. + /// No prev ⇒ BadNoData, null value. + /// prev Bad (incl. an Empty outage marker) ⇒ t lies inside an outage: null value + /// carrying prev's Bad status code (info bits stripped) — never better than its source. + /// prev usable and next Bad ⇒ prev's value held (stepped, NOT interpolated into the + /// outage) with UncertainDataSubNormal + Interpolated bits. + /// Double/Int64 with a usable next ⇒ linear interpolation (Int64 rounded to the nearest integer, + /// ties away from zero, keeping the type); status Good + Interpolated when both neighbours are Good, + /// else UncertainDataSubNormal + Interpolated. + /// Boolean/String/DateTime, or no next (after the last sample) ⇒ stepped: prev's value with + /// prev's status code + Interpolated bits — except that a time later than now that is stepped from the + /// last sample is UncertainDataSubNormal + Interpolated. + /// + /// + /// Requested time. + /// The stored sample at or before , if any. + /// The stored sample after , if any. + /// Server timestamp for the result. + /// The current time (for the beyond-now rule). + /// The value at . + public static DataValueSnapshot Evaluate(DateTime t, StoredSample? prev, StoredSample? next, DateTime serverTime, DateTime now) + { + if (prev is not { } p) + return new DataValueSnapshot(null, HistorianStatusCodes.BadNoData, t, serverTime); + if (p.TimeUtc == t) + return new DataValueSnapshot(p.Value.ToObject(), p.Status, t, serverTime); + + // Inside an outage: report the outage's own Bad code, never an upgraded Uncertain. + if (!p.IsUsable) + return new DataValueSnapshot(null, p.Status & StatusBits.CodeBitsMask, t, serverTime); + + // Usable previous, Bad next: hold the previous value up to the outage; do not interpolate into it. + if (next is { IsUsable: false }) + return new DataValueSnapshot(p.Value.ToObject(), + HistorianInfoBits.WithInterpolated(HistorianStatusCodes.UncertainDataSubNormal), t, serverTime); + + if (next is { } n && p.Value.Kind is HistorianValueKind.Double or HistorianValueKind.Int64 + && n.Value.TryGetDouble(out var nv)) + { + var fraction = (double)(t.Ticks - p.TimeUtc.Ticks) / (n.TimeUtc.Ticks - p.TimeUtc.Ticks); + var v = p.Value.Number + (nv - p.Value.Number) * fraction; + object value = p.Value.Kind == HistorianValueKind.Int64 + ? (long)Math.Round(v, MidpointRounding.AwayFromZero) + : v; + var status = p.IsGood && n.IsGood ? HistorianStatusCodes.Good : HistorianStatusCodes.UncertainDataSubNormal; + return new DataValueSnapshot(value, HistorianInfoBits.WithInterpolated(status), t, serverTime); + } + + // Beyond the last sample AND beyond now: the value is held into the future, so it is not known to be Good. + if (next is null && t > now) + return new DataValueSnapshot(p.Value.ToObject(), + HistorianInfoBits.WithInterpolated(HistorianStatusCodes.UncertainDataSubNormal), t, serverTime); + + // Stepped (non-numeric kind, or after the last sample): hold the previous value and its status. + return new DataValueSnapshot(p.Value.ToObject(), HistorianInfoBits.WithInterpolated(p.Status), t, serverTime); + } +} diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/BootstrapFailureCache.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/BootstrapFailureCache.cs new file mode 100644 index 00000000..2dfd1f67 --- /dev/null +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/BootstrapFailureCache.cs @@ -0,0 +1,46 @@ +using System.Runtime.ExceptionServices; + +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common; + +/// +/// The negative cache a schema bootstrapper keeps for its last failure: within the cache window the failure is +/// re-thrown without touching the database, so a missing schema (or an outage) does not serialise every write, read +/// and alarm event behind one semaphore and one round-trip each. A failure is NOT latched: after the window the +/// caller retries. Not thread-safe; the caller guards it (the bootstrapper's gate). +/// +public sealed class BootstrapFailureCache +{ + /// How long the last bootstrap failure is re-thrown before the database is tried again (5 s). + public static readonly TimeSpan DefaultDuration = TimeSpan.FromSeconds(5); + + private readonly TimeProvider _clock; + private readonly TimeSpan _duration; + private ExceptionDispatchInfo? _lastFailure; + private DateTimeOffset _retryAfter; + + /// Creates the cache. + /// Clock for the cache window. + /// How long a recorded failure is re-thrown (normally ). + public BootstrapFailureCache(TimeProvider clock, TimeSpan duration) + { + _clock = clock ?? throw new ArgumentNullException(nameof(clock)); + _duration = duration; + } + + /// Re-throws the last recorded failure (original stack preserved) while its window is open. + public void ThrowIfCached() + { + if (_lastFailure is { } cached && _clock.GetUtcNow() < _retryAfter) cached.Throw(); + } + + /// Records a failure; it is re-thrown by for the cache window. + /// The bootstrap failure. + public void Record(Exception ex) + { + _lastFailure = ExceptionDispatchInfo.Capture(ex); + _retryAfter = _clock.GetUtcNow() + _duration; + } + + /// Forgets any recorded failure (after a success). + public void Clear() => _lastFailure = null; +} diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/BucketPlan.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/BucketPlan.cs new file mode 100644 index 00000000..29c95bd1 --- /dev/null +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/BucketPlan.cs @@ -0,0 +1,91 @@ +using ZB.MOM.WW.OtOpcUa.Core.Abstractions; + +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common; + +/// +/// The bucket layout of a processed (aggregate) read. Bucket i = [start + i·interval, +/// start + (i+1)·interval) (start-inclusive, end-exclusive to the tick; the last bucket is truncated at +/// end). An interval <= 0 ⇒ one interval [start, end). start == end ⇒ no intervals. +/// end < start (reverse) ⇒ the window is swapped, computed forward, and assembled in descending bucket +/// order (buckets stay anchored at the earlier time). +/// +public sealed class BucketPlan +{ + /// Spec §9.7: a processed read covering more intervals than this is refused. + public const int MaxBuckets = 100_000; + + private BucketPlan(DateTime startUtc, DateTime endUtc, bool reversed, long intervalTicks, long bucketCount) + { + StartUtc = startUtc; + EndUtc = endUtc; + Reversed = reversed; + IntervalTicks = intervalTicks; + BucketCount = bucketCount; + } + + /// The earlier end of the window (the forward start). + public DateTime StartUtc { get; } + + /// The later end of the window (the forward, exclusive end). + public DateTime EndUtc { get; } + + /// True when the request came in reversed (end < start). + public bool Reversed { get; } + + /// Bucket width in 100 ns ticks (clamped to the window; 0 for an empty window). + public long IntervalTicks { get; } + + /// Number of buckets (0 for an empty window). + public long BucketCount { get; } + + /// as 100 ns ticks since the Unix epoch. + public long StartUnixTicks => (StartUtc - DateTime.UnixEpoch).Ticks; + + /// Plans the buckets of a processed read. + /// Requested start. + /// Requested end (earlier than start for a reverse read). + /// Bucket width; <= 0 means one bucket over the whole window. + /// The plan. + /// The window holds more than intervals + /// (parameter name interval). + public static BucketPlan Create(DateTime startUtc, DateTime endUtc, TimeSpan interval) + { + var reverse = endUtc < startUtc; + var (start, end) = reverse ? (endUtc, startUtc) : (startUtc, endUtc); + var windowTicks = end.Ticks - start.Ticks; + // Everything stays in 100 ns ticks (≤ ≈3.2e18 across 0001–9999, inside a long) — no ns scaling that could + // wrap. An interval wider than the window is one bucket, so clamp it to the window; the ceiling division + // is written without "+ interval - 1" so a TimeSpan near MaxValue cannot overflow either. + var intervalTicks = interval > TimeSpan.Zero ? Math.Min(interval.Ticks, windowTicks) : windowTicks; + long bucketCount = windowTicks == 0 ? 0 : windowTicks / intervalTicks + (windowTicks % intervalTicks == 0 ? 0 : 1); + if (bucketCount > MaxBuckets) + throw new ArgumentOutOfRangeException(nameof(interval), bucketCount, + $"A processed read may cover at most {MaxBuckets} intervals; this window holds {bucketCount}."); + return new BucketPlan(start, end, reverse, intervalTicks, bucketCount); + } + + /// The start time of bucket (UTC kind). + /// Zero-based bucket index. + /// The bucket start. + public DateTime BucketStart(long index) => + DateTime.SpecifyKind(new DateTime(StartUtc.Ticks + index * IntervalTicks), DateTimeKind.Utc); + + /// One snapshot per bucket in window order; missing buckets get ; reversed if the window was. + /// Evaluated snapshots of the non-empty buckets, by bucket index. + /// Builds the snapshot of a bucket with no row, from its start time. + /// snapshots. + public IReadOnlyList Assemble(IReadOnlyDictionary byBucket, + Func empty) + { + ArgumentNullException.ThrowIfNull(byBucket); + ArgumentNullException.ThrowIfNull(empty); + var result = new List((int)BucketCount); + for (long i = 0; i < BucketCount; i++) + { + var bucketStart = BucketStart(i); + result.Add(byBucket.TryGetValue(i, out var v) ? v : empty(bucketStart)); + } + if (Reversed) result.Reverse(); + return result; + } +} diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/BucketRow.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/BucketRow.cs new file mode 100644 index 00000000..2f178a39 --- /dev/null +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/BucketRow.cs @@ -0,0 +1,57 @@ +using ZB.MOM.WW.OtOpcUa.Core.Abstractions; + +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common; + +/// +/// Per-bucket statistics one aggregate query row produces (one non-empty bucket). Every backend's SQL fills the same +/// fields; turns them into the OPC UA value and status. +/// +/// +/// Sample classes: usable = severity Good or Uncertain AND a stored value (an Empty sample is an outage marker +/// and counts as Bad whatever its code); good = usable with severity Good; bad = not usable. +/// Every value field is computed over usable samples only. +/// +public sealed class BucketRow +{ + /// Zero-based bucket index within the window. + public long Bucket { get; set; } + + /// All samples in the bucket. + public int Total { get; set; } + + /// Usable samples. + public int Usable { get; set; } + + /// Good samples. + public int Good { get; set; } + + /// Bad (not usable) samples. + public int Bad { get; set; } + + /// Usable String / DateTime samples. + public int NonNumeric { get; set; } + + /// Average of the usable numeric values (null when none). + public double? Avg { get; set; } + + /// Minimum of the usable numeric values (null when none). + public double? Min { get; set; } + + /// Maximum of the usable numeric values (null when none). + public double? Max { get; set; } + + /// Sum of the usable numeric values (null when none). + public double? Sum { get; set; } + + /// First usable sample's value by time (Empty when none). + public HistorianValue First { get; set; } + + /// Last usable sample's value by time (Empty when none). + public HistorianValue Last { get; set; } + + /// Extreme (min or max) usable numeric value (null when none). + public double? Extreme { get; set; } + + /// Time of (ties ⇒ earliest; null when none). + public DateTime? ExtremeTime { get; set; } +} diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/EventContinuation.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/EventContinuation.cs new file mode 100644 index 00000000..5e3ef7cb --- /dev/null +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/EventContinuation.cs @@ -0,0 +1,11 @@ +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common; + +/// The opaque Events continuation point of the SQL historian backends (only its presence is contractual). +public static class EventContinuation +{ + /// Encodes the first not-returned event as "{EventTimeUtc.Ticks}|{EventId}". + /// The event's time. + /// The event's id. + /// The continuation text (callers send its UTF-8 bytes). + public static string Encode(DateTime eventTimeUtc, string eventId) => $"{eventTimeUtc.Ticks}|{eventId}"; +} diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlHistorianDataTypes.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/HistorianDataTypes.cs similarity index 72% rename from src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlHistorianDataTypes.cs rename to src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/HistorianDataTypes.cs index 90dbdb53..8e1787bb 100644 --- a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlHistorianDataTypes.cs +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/HistorianDataTypes.cs @@ -1,18 +1,18 @@ using ZB.MOM.WW.OtOpcUa.Core.Abstractions; -namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer; +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common; /// -/// The one place that maps value kinds to Tag.DataType (the schema stores a byte). +/// The one place that maps value kinds to the stored tag data type (a byte). /// Used by the value writer and the updater only when they have to create a tag row on a miss; provisioning /// later corrects it. /// -internal static class SqlHistorianDataTypes +public static class HistorianDataTypes { /// Data type used when a tag is created from a batch that holds no typed value. public const DriverDataType DefaultForEmpty = DriverDataType.Float64; - /// Maps a value kind to the stored Tag.DataType byte (Empty ⇒ ). + /// Maps a value kind to the stored tag data-type byte (Empty ⇒ ). /// Value kind. /// The as a byte. public static byte FromKind(HistorianValueKind kind) => (byte)(kind switch diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/HistorianHealthCounters.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/HistorianHealthCounters.cs new file mode 100644 index 00000000..1519b971 --- /dev/null +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/HistorianHealthCounters.cs @@ -0,0 +1,70 @@ +using ZB.MOM.WW.OtOpcUa.Core.Abstractions; + +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common; + +/// +/// The read-side health counters a SQL historian data source reports through +/// . Lock-guarded; safe to call from concurrent reads. +/// +public sealed class HistorianHealthCounters +{ + private readonly object _lock = new(); + private DateTime? _lastSuccessUtc; + private DateTime? _lastFailureUtc; + private string? _lastError; + private long _totalQueries; + private long _totalSuccesses; + private long _totalFailures; + private int _consecutiveFailures; + + /// Records a successful query. + public void RecordSuccess() + { + lock (_lock) + { + _totalQueries++; + _totalSuccesses++; + _consecutiveFailures = 0; + _lastSuccessUtc = DateTime.UtcNow; + } + } + + /// Records a failed query. An (deadline / client abort) is not a backend fault and is ignored. + /// The failure. + public void RecordFailure(Exception ex) + { + ArgumentNullException.ThrowIfNull(ex); + if (ex is OperationCanceledException) return; + lock (_lock) + { + _totalQueries++; + _totalFailures++; + _consecutiveFailures++; + _lastFailureUtc = DateTime.UtcNow; + _lastError = $"{ex.GetType().Name}: {ex.Message}"; + } + } + + /// The current counters. The connection is reported open when there has been a success and no failure since. + /// The snapshot. + public HistorianHealthSnapshot Snapshot() + { + lock (_lock) + { + var open = _totalSuccesses > 0 && _consecutiveFailures == 0; + return new HistorianHealthSnapshot( + TotalQueries: _totalQueries, + TotalSuccesses: _totalSuccesses, + TotalFailures: _totalFailures, + ConsecutiveFailures: _consecutiveFailures, + LastSuccessTime: _lastSuccessUtc, + LastFailureTime: _lastFailureUtc, + LastError: _lastError, + ProcessConnectionOpen: open, + EventConnectionOpen: open, + ActiveProcessNode: null, + ActiveEventNode: null, + Nodes: []); + } + } +} diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/HistorianInfoBits.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/HistorianInfoBits.cs new file mode 100644 index 00000000..1a067546 --- /dev/null +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/HistorianInfoBits.cs @@ -0,0 +1,32 @@ +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common; + +/// Part 4 §7.39 historian info bits OR-ed onto a status code. +public static class HistorianInfoBits +{ + /// InfoType = DataValue. + public const uint InfoTypeDataValue = 0x00000400u; + + /// The value was produced by an aggregate calculation. + public const uint HistorianCalculated = 0x1u; + + /// The value was interpolated. + public const uint HistorianInterpolated = 0x2u; + + /// + /// Marks as a Calculated data value. Any historian bits the status already carries + /// (e.g. a stored Interpolated sample) are cleared first, so the result is never both. + /// + /// Base status code. + /// Status with InfoType and Calculated bits set. + public static uint WithCalculated(uint status) => + (status & ~StatusBits.HistorianBitsMask) | InfoTypeDataValue | HistorianCalculated; + + /// + /// Marks as an Interpolated data value. Any historian bits the status already + /// carries (e.g. a stored Calculated sample) are cleared first, so the result is never both. + /// + /// Base status code. + /// Status with InfoType and Interpolated bits set. + public static uint WithInterpolated(uint status) => + (status & ~StatusBits.HistorianBitsMask) | InfoTypeDataValue | HistorianInterpolated; +} diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/HistorianSampleRow.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/HistorianSampleRow.cs new file mode 100644 index 00000000..cc37c856 --- /dev/null +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/HistorianSampleRow.cs @@ -0,0 +1,9 @@ +using ZB.MOM.WW.OtOpcUa.Core.Abstractions; + +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common; + +/// One sample bound for storage. +/// Source timestamp (UTC, at the backend's resolution). +/// Typed value. +/// OPC UA status code (stored bit-for-bit). +public readonly record struct HistorianSampleRow(DateTime TimeUtc, HistorianValue Value, uint StatusCode); diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/HistorianSchemaMissingException.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/HistorianSchemaMissingException.cs new file mode 100644 index 00000000..f0955e94 --- /dev/null +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/HistorianSchemaMissingException.cs @@ -0,0 +1,9 @@ +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common; + +/// Thrown when AutoCreateSchema=false and the historian schema is absent or behind. +public sealed class HistorianSchemaMissingException : Exception +{ + /// Creates the exception. + /// Message naming the missing version. + public HistorianSchemaMissingException(string message) : base(message) { } +} diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/HistorianStatusCodes.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/HistorianStatusCodes.cs new file mode 100644 index 00000000..e8896f66 --- /dev/null +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/HistorianStatusCodes.cs @@ -0,0 +1,27 @@ +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common; + +/// +/// Every OPC UA status the SQL historian backends emit, as named constants (inline literals are invisible to +/// StatusCodeParityTests). Values mirror Opc.Ua.StatusCodes. +/// +public static class HistorianStatusCodes +{ + public const uint Good = 0x00000000u; + public const uint GoodEntryInserted = 0x00A20000u; + public const uint GoodEntryReplaced = 0x00A30000u; + public const uint BadNoData = 0x809B0000u; + public const uint BadEntryExists = 0x809F0000u; + public const uint BadNoEntryExists = 0x80A00000u; + public const uint BadInvalidTimestamp = 0x80230000u; + public const uint BadAggregateInvalidInputs = 0x80D60000u; + public const uint UncertainDataSubNormal = 0x40A40000u; + + /// Stored for a non-finite Double (NaN / ±Infinity) that arrived with a non-Bad status. + public const uint BadOutOfRange = 0x803C0000u; + + /// + /// A HistoryUpdate value that cannot be stored: a Double NaN / ±Infinity, an Empty (null) value with a non-Bad + /// status, or a tag name longer than . + /// + public const uint BadInvalidArgument = 0x80AB0000u; +} diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/HistorianTagNames.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/HistorianTagNames.cs new file mode 100644 index 00000000..aa3a1b73 --- /dev/null +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/HistorianTagNames.cs @@ -0,0 +1,17 @@ +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common; + +/// +/// Historian tag-name rules. Names longer than are never truncated: a sized parameter +/// would silently cut them and resolve them onto whatever tag owns the first characters, so +/// they are refused (writes) or answered as not found (reads). +/// +public static class HistorianTagNames +{ + /// Longest storable tag (and event source) name; longer names are refused. + public const int MaxLength = 400; + + /// True when is storable without truncation. + /// Historian tag name. + /// Whether the name is non-null and at most characters. + public static bool IsValid(string? name) => name is not null && name.Length <= MaxLength; +} diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/HistoricalEventRows.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/HistoricalEventRows.cs new file mode 100644 index 00000000..70de0a4b --- /dev/null +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/HistoricalEventRows.cs @@ -0,0 +1,10 @@ +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common; + +/// Helpers for turning stored event rows into HistoricalEvents. +public static class HistoricalEventRows +{ + /// Clamps a stored severity into the range the event carries. + /// Stored severity. + /// The severity clamped to 0..65535. + public static ushort ClampSeverity(int severity) => (ushort)Math.Clamp(severity, 0, ushort.MaxValue); +} diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/HistoryUpdatePlanner.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/HistoryUpdatePlanner.cs new file mode 100644 index 00000000..3e9c27db --- /dev/null +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/HistoryUpdatePlanner.cs @@ -0,0 +1,175 @@ +using ZB.MOM.WW.OtOpcUa.Core.Abstractions; + +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common; + +/// What the backend did with one HistoryUpdate winner row. +public enum UpsertOutcome +{ + /// The row was absent and was inserted. + Inserted, + + /// The row existed and was replaced. + Replaced, +} + +/// The database-independent half of one UpdateDataAsync call (see ). +public sealed class UpdatePlan +{ + internal UpdatePlan(uint?[] results, IReadOnlyList<(int Index, HistorianSampleRow Row)> winners, + long?[] keys, IReadOnlyDictionary winnerByTick) + { + Results = results; + Winners = winners; + Keys = keys; + WinnerByTick = winnerByTick; + } + + /// Pre-decided codes (invalid ts, NaN, superseded duplicate…); null = decided by the backend. + public uint?[] Results { get; } + + /// Rows to send, de-duplicated per mode, in input order. + public IReadOnlyList<(int Index, HistorianSampleRow Row)> Winners { get; } + + /// Per input item: its (truncated) timestamp ticks, or null when the timestamp is invalid. + internal long?[] Keys { get; } + + /// (Truncated) tick → input index of the item actually sent. + internal IReadOnlyDictionary WinnerByTick { get; } +} + +/// +/// Per-item result rules for OPC UA HistoryUpdate (spec §4 "Updates"): exactly one status per input item, in input +/// order. +/// Duplicate timestamps within one request (compared at the backend's resolution): Insert — the first +/// occurrence is applied, later ones are BadEntryExists; Replace/Update — last wins, and every earlier +/// occurrence of that timestamp reports the same code as the winning one. DeleteAtTime — the first occurrence +/// reports the real outcome, later ones BadNoEntryExists. +/// Invalid items. A null or timestamp is BadInvalidTimestamp. +/// A Double NaN/±Infinity is rejected with BadInvalidArgument. An Empty (null) value is only accepted with a +/// Bad status (an operator-authored outage marker); with a Good or Uncertain status it is BadInvalidArgument. +/// Other items still proceed. +/// Tag names over characters are refused, never truncated: +/// every item reports BadInvalidArgument and nothing is sent. +/// +public static class HistoryUpdatePlanner +{ + /// The pre-pass: classifies every item and picks the winner row per timestamp. + /// Historian tag name. + /// Insert, Replace or Update. + /// Input items, in request order. + /// 1 = 100 ns (SQL Server); 10 = timestamps truncated to µs BEFORE de-duplication (QuestDB). + /// The plan. + public static UpdatePlan Plan(string tag, HistoryUpdateMode mode, IReadOnlyList values, + long resolutionTicks = 1) + { + ArgumentNullException.ThrowIfNull(tag); + ArgumentNullException.ThrowIfNull(values); + ArgumentOutOfRangeException.ThrowIfLessThan(resolutionTicks, 1); + var results = new uint?[values.Count]; + var winner = new Dictionary(); // tick -> index of the item actually sent + if (!HistorianTagNames.IsValid(tag)) + { + Array.Fill(results, HistorianStatusCodes.BadInvalidArgument); + return new UpdatePlan(results, [], new long?[values.Count], winner); + } + var times = new DateTime[values.Count]; + var keys = new long?[values.Count]; + for (var i = 0; i < values.Count; i++) + { + var v = values[i]; + if (v.TimestampUtc is not { } ts || ts == DateTime.MinValue) + { + results[i] = HistorianStatusCodes.BadInvalidTimestamp; + continue; + } + times[i] = LiveWritePreparer.Truncate(ts, resolutionTicks); + keys[i] = times[i].Ticks; + if ((v.Value.Kind == HistorianValueKind.Double && !double.IsFinite(v.Value.Number)) + || (v.Value.Kind == HistorianValueKind.Empty && !StatusBits.IsBad(v.StatusCode))) + { + results[i] = HistorianStatusCodes.BadInvalidArgument; + continue; + } + var key = times[i].Ticks; + if (!winner.TryGetValue(key, out _)) + winner[key] = i; + else if (mode == HistoryUpdateMode.Insert) + results[i] = HistorianStatusCodes.BadEntryExists; // first occurrence stays the winner + else + winner[key] = i; // last wins + } + + var send = winner.Values.OrderBy(i => i) + .Select(i => (i, new HistorianSampleRow(times[i], values[i].Value, values[i].StatusCode))) + .ToList(); + return new UpdatePlan(results, send, keys, winner); + } + + /// The post-pass: fills in each winner's code from what the backend wrote, then the superseded duplicates. + /// The plan from . + /// The same mode passed to . + /// Ticks of each winner that the backend actually wrote → Inserted/Replaced; absent ⇒ not + /// written (Insert: the row already existed ⇒ BadEntryExists; Replace: no row ⇒ BadNoEntryExists). + /// One status per input item, in input order. + public static IReadOnlyList Complete(UpdatePlan plan, HistoryUpdateMode mode, + IReadOnlyDictionary outcomeByTick) + { + ArgumentNullException.ThrowIfNull(plan); + ArgumentNullException.ThrowIfNull(outcomeByTick); + var results = (uint?[])plan.Results.Clone(); + + // No written row means the absent branch: Insert => matched (exists); Replace => not matched. + var absent = mode == HistoryUpdateMode.Replace + ? HistorianStatusCodes.BadNoEntryExists + : HistorianStatusCodes.BadEntryExists; + foreach (var (index, row) in plan.Winners) + results[index] = outcomeByTick.TryGetValue(row.TimeUtc.Ticks, out var o) + ? o == UpsertOutcome.Inserted ? HistorianStatusCodes.GoodEntryInserted : HistorianStatusCodes.GoodEntryReplaced + : absent; + + var final = new uint[results.Length]; + for (var i = 0; i < results.Length; i++) final[i] = results[i] ?? 0u; + + // Superseded earlier duplicates (Replace/Update) share the winner's code. + if (mode != HistoryUpdateMode.Insert) + for (var i = 0; i < results.Length; i++) + if (plan.Keys[i] is { } key && results[i] is null + && plan.WinnerByTick.TryGetValue(key, out var w) && w != i) + final[i] = final[w]; + return final; + } + + /// + /// DeleteAtTime results: ⇒ BadInvalidTimestamp; the first occurrence of a + /// deleted time ⇒ Good; everything else (unknown tag, no row, later duplicate) ⇒ BadNoEntryExists. + /// + /// Requested times, in request order (already at the backend's resolution). + /// Ticks of the rows the backend actually deleted. + /// Whether the tag exists. + /// One status per requested time. + public static IReadOnlyList DeleteAtTimeResults(IReadOnlyList times, ISet deletedTicks, bool tagKnown) + { + ArgumentNullException.ThrowIfNull(times); + ArgumentNullException.ThrowIfNull(deletedTicks); + var results = new uint[times.Count]; + var first = new Dictionary(); + for (var i = 0; i < times.Count; i++) + { + results[i] = HistorianStatusCodes.BadNoEntryExists; + if (times[i] == DateTime.MinValue) results[i] = HistorianStatusCodes.BadInvalidTimestamp; + else first.TryAdd(times[i].Ticks, i); + } + if (!tagKnown) return results; + foreach (var (ticks, idx) in first) + if (deletedTicks.Contains(ticks)) + results[idx] = HistorianStatusCodes.Good; + return results; + } + + /// DeleteRaw result: unknown tag or nothing deleted ⇒ BadNoData; otherwise Good. + /// Whether the tag exists. + /// Rows the backend deleted. + /// The status. + public static uint DeleteRawResult(bool tagKnown, long rowsAffected) => + tagKnown && rowsAffected > 0 ? HistorianStatusCodes.Good : HistorianStatusCodes.BadNoData; +} diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/LiveWritePreparer.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/LiveWritePreparer.cs new file mode 100644 index 00000000..d10faa74 --- /dev/null +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/LiveWritePreparer.cs @@ -0,0 +1,54 @@ +using ZB.MOM.WW.OtOpcUa.Core.Abstractions; + +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common; + +/// +/// Normalises a live-value batch for storage. Non-finite doubles. NaN / ±Infinity are stored as +/// so the sample's presence survives. An Empty sample is always Bad: +/// a non-finite Double whose status is not already Bad is stored with BadOutOfRange; any other Empty with a +/// non-Bad status is stored with BadNoData; an existing Bad code is kept. Duplicate timestamps within +/// a batch collapse to the first (matching insert-if-absent across batches). +/// +public static class LiveWritePreparer +{ + /// + /// Normalises a batch: null timestamp ⇒ now, timestamps truncated to , + /// non-finite doubles ⇒ Empty, every Empty forced Bad, and duplicate (truncated) timestamps collapsed to the first. + /// + /// Input samples, in append order. + /// Clock for null timestamps. + /// 1 = keep 100 ns (SQL Server); 10 = truncate to µs BEFORE de-duplication (QuestDB). + /// Time-unique rows. + public static List Prepare(IReadOnlyList values, Func utcNow, + long resolutionTicks = 1) + { + ArgumentOutOfRangeException.ThrowIfLessThan(resolutionTicks, 1); + var seen = new HashSet(); + var rows = new List(values.Count); + foreach (var v in values) + { + var t = v.TimestampUtc ?? utcNow(); + t = t.Kind == DateTimeKind.Local ? t.ToUniversalTime() : DateTime.SpecifyKind(t, DateTimeKind.Utc); + t = Truncate(t, resolutionTicks); + if (!seen.Add(t.Ticks)) continue; + var value = v.Value; + var status = v.StatusCode; + if (value.Kind == HistorianValueKind.Double && !double.IsFinite(value.Number)) + { + value = HistorianValue.Empty; + if (!StatusBits.IsBad(status)) status = HistorianStatusCodes.BadOutOfRange; + } + else if (value.Kind == HistorianValueKind.Empty && !StatusBits.IsBad(status)) + { + status = HistorianStatusCodes.BadNoData; + } + rows.Add(new HistorianSampleRow(t, value, status)); + } + + return rows; + } + + /// Truncates down to a multiple of , keeping its kind. + internal static DateTime Truncate(DateTime t, long resolutionTicks) => + resolutionTicks == 1 ? t : t.AddTicks(-(t.Ticks % resolutionTicks)); +} diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/SampleColumns.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/SampleColumns.cs new file mode 100644 index 00000000..f8ee6bb1 --- /dev/null +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/SampleColumns.cs @@ -0,0 +1,66 @@ +using ZB.MOM.WW.OtOpcUa.Core.Abstractions; + +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common; + +/// The (ValueKind, NumValue, IntValue, StrValue) column quad for a value. +/// The as a byte. +/// Numeric column (null = NULL). +/// Integer column (null = NULL). +/// String column (null = NULL). +public readonly record struct SampleColumns(byte ValueKind, double? NumValue, long? IntValue, string? StrValue); + +/// +/// HistorianValue ⇄ column mapping (spec §4). Mapping: Double → Num; Int64 → Int + Num(double); +/// Boolean → Int 0/1 + Num 0/1; String → Str; DateTime → Int (ticks); Empty → all NULL. +/// +public static class SampleColumnMapper +{ + /// Splits a value into its column quad. + /// Typed value. + /// The columns to store. + public static SampleColumns ToColumns(HistorianValue v) => v.Kind switch + { + HistorianValueKind.Double => new((byte)v.Kind, v.Number, null, null), + HistorianValueKind.Int64 => new((byte)v.Kind, v.Number, v.Integer, null), + HistorianValueKind.Boolean => new((byte)v.Kind, v.Integer != 0 ? 1d : 0d, v.Integer != 0 ? 1L : 0L, null), + HistorianValueKind.String => new((byte)v.Kind, null, null, v.Text ?? ""), + HistorianValueKind.DateTime => new((byte)v.Kind, null, v.Integer, null), + _ => new((byte)HistorianValueKind.Empty, null, null, null), + }; + + /// Rebuilds a value from stored columns. + /// ValueKind column. + /// NumValue column (null = NULL). + /// IntValue column (null = NULL). + /// StrValue column (null = NULL). + /// The typed value; an unknown kind yields . + public static HistorianValue FromColumns(byte valueKind, double? num, long? integer, string? str) => (HistorianValueKind)valueKind switch + { + HistorianValueKind.Double => HistorianValue.FromDouble(num ?? 0d), + HistorianValueKind.Int64 => HistorianValue.FromInt64(integer ?? 0L), + HistorianValueKind.Boolean => HistorianValue.FromBoolean((integer ?? 0L) != 0), + HistorianValueKind.String => HistorianValue.FromString(str ?? ""), + HistorianValueKind.DateTime => new HistorianValue(HistorianValueKind.DateTime, 0d, integer ?? 0L, null), + _ => HistorianValue.Empty, + }; + + /// Stores a uint status bit-for-bit in an int column. + /// Status code. + /// Same bits as int. + public static int ToDbStatus(uint status) => unchecked((int)status); + + /// Reads an int status column back to a uint code. + /// Column value. + /// Same bits as uint. + public static uint FromDbStatus(int dbStatus) => unchecked((uint)dbStatus); + + /// Stores a uint status in a 64-bit column (zero-extended, so it is never negative). + /// Status code. + /// The status as a non-negative long. + public static long ToLongStatus(uint status) => status; + + /// Reads a 64-bit status column back to a uint code (low 32 bits). + /// Column value. + /// The status code. + public static uint FromLongStatus(long dbStatus) => unchecked((uint)dbStatus); +} diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/StatusBits.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/StatusBits.cs new file mode 100644 index 00000000..cf276ce2 --- /dev/null +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/StatusBits.cs @@ -0,0 +1,33 @@ +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common; + +/// +/// Bit masks over a 32-bit OPC UA status code (Part 4 §7.39). These are masks, not emitted codes, so their +/// names deliberately avoid the Good/Bad/Uncertain prefix the status-code parity scan keys on. +/// +public static class StatusBits +{ + /// Severity + sub-code (top 16 bits) of a status, i.e. the code without info bits. + public const uint CodeBitsMask = 0xFFFF0000u; + + /// The two severity bits (Good = 00, Uncertain = 01, Bad = 10). + public const uint SeverityMask = 0xC0000000u; + + /// The severity bit set on every Bad code. + public const uint SeverityBadBit = 0x80000000u; + + /// The severity bit set on every Uncertain code (bit 30). + public const uint SeverityUncertainBit = 0x40000000u; + + /// The two historian info bits (Calculated, Interpolated) in the low bits of the info field. + public const uint HistorianBitsMask = 0x3u; + + /// True when has Bad severity. + /// A status code. + /// Whether the Bad severity bit is set. + public static bool IsBad(uint status) => (status & SeverityBadBit) != 0; + + /// True when has Good severity. + /// A status code. + /// Whether both severity bits are clear. + public static bool IsGood(uint status) => (status & SeverityMask) == 0; +} diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/StoredSample.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/StoredSample.cs new file mode 100644 index 00000000..b155d4bc --- /dev/null +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/StoredSample.cs @@ -0,0 +1,32 @@ +using ZB.MOM.WW.OtOpcUa.Core.Abstractions; + +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common; + +/// A stored sample as read back: source time, typed value and stored status. +/// Source timestamp (UTC). +/// Typed value ( for an outage marker). +/// Stored OPC UA status code. +public readonly record struct StoredSample(DateTime TimeUtc, HistorianValue Value, uint Status) +{ + /// Severity Good or Uncertain, and a value present (an Empty sample is an outage marker). + public bool IsUsable => !StatusBits.IsBad(Status) && Value.Kind != HistorianValueKind.Empty; + + /// Usable with severity Good. + public bool IsGood => IsUsable && StatusBits.IsGood(Status); + + /// + /// Builds a sample from stored columns, applying the quality clamp: an Empty (no-value) row whose stored status + /// is not Bad is served as BadNoData — defence in depth for rows written before the writer forced every + /// Empty sample Bad. + /// + /// Source timestamp (UTC). + /// Stored value. + /// Stored status code. + /// The sample, clamped. + public static StoredSample FromStored(DateTime timeUtc, HistorianValue value, uint status) + { + if (value.Kind == HistorianValueKind.Empty && !StatusBits.IsBad(status)) + status = HistorianStatusCodes.BadNoData; + return new StoredSample(timeUtc, value, status); + } +} diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.csproj b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.csproj new file mode 100644 index 00000000..8e0cb860 --- /dev/null +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.csproj @@ -0,0 +1,22 @@ + + + + true + true + $(NoWarn);CS1591 + AnyCPU;x64 + ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common + + + + + + + + + + diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SampleRowMapper.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SampleRowMapper.cs index e9e20ac2..49ba2ccf 100644 --- a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SampleRowMapper.cs +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SampleRowMapper.cs @@ -1,22 +1,12 @@ using System.Data; using Microsoft.Data.SqlClient.Server; -using ZB.MOM.WW.OtOpcUa.Core.Abstractions; +using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common; namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer; -/// One sample bound for a SampleRow TVP. -/// Source timestamp (UTC, full tick precision). -/// Typed value. -/// OPC UA status code (stored bit-for-bit as int). -internal readonly record struct SqlSampleRow(DateTime TimeUtc, HistorianValue Value, uint StatusCode); - -/// The (ValueKind, NumValue, IntValue, StrValue) column quad for a value. -internal readonly record struct SampleColumns(byte ValueKind, double? NumValue, long? IntValue, string? StrValue); - /// -/// HistorianValue ⇄ column mapping (spec §4) and the SqlDataRecord TVP builders. Mapping: -/// Double → Num; Int64 → Int + Num(double); Boolean → Int 0/1 + Num 0/1; String → Str; DateTime → Int (ticks); -/// Empty → all NULL. +/// The SqlDataRecord TVP builders. The HistorianValue ⇄ column mapping itself is +/// (shared with the other SQL backends). /// internal static class SampleRowMapper { @@ -32,60 +22,21 @@ internal static class SampleRowMapper private static readonly SqlMetaData[] TimeMeta = [new("T", SqlDbType.DateTime2, 0, 7)]; - /// Splits a value into its column quad. - /// Typed value. - /// The columns to store. - public static SampleColumns ToColumns(HistorianValue v) => v.Kind switch - { - HistorianValueKind.Double => new((byte)v.Kind, v.Number, null, null), - HistorianValueKind.Int64 => new((byte)v.Kind, v.Number, v.Integer, null), - HistorianValueKind.Boolean => new((byte)v.Kind, v.Integer != 0 ? 1d : 0d, v.Integer != 0 ? 1L : 0L, null), - HistorianValueKind.String => new((byte)v.Kind, null, null, v.Text ?? ""), - HistorianValueKind.DateTime => new((byte)v.Kind, null, v.Integer, null), - _ => new((byte)HistorianValueKind.Empty, null, null, null), - }; - - /// Rebuilds a value from stored columns. - /// ValueKind column. - /// NumValue column (null = NULL). - /// IntValue column (null = NULL). - /// StrValue column (null = NULL). - /// The typed value; an unknown kind yields . - public static HistorianValue FromColumns(byte valueKind, double? num, long? integer, string? str) => (HistorianValueKind)valueKind switch - { - HistorianValueKind.Double => HistorianValue.FromDouble(num ?? 0d), - HistorianValueKind.Int64 => HistorianValue.FromInt64(integer ?? 0L), - HistorianValueKind.Boolean => HistorianValue.FromBoolean((integer ?? 0L) != 0), - HistorianValueKind.String => HistorianValue.FromString(str ?? ""), - HistorianValueKind.DateTime => new HistorianValue(HistorianValueKind.DateTime, 0d, integer ?? 0L, null), - _ => HistorianValue.Empty, - }; - - /// Stores a uint status bit-for-bit in the int column. - /// Status code. - /// Same bits as int. - public static int ToDbStatus(uint status) => unchecked((int)status); - - /// Reads the int status column back to a uint code. - /// Column value. - /// Same bits as uint. - public static uint FromDbStatus(int dbStatus) => unchecked((uint)dbStatus); - /// Builds SampleRow TVP records (datetime2(7); never a DataTable). Lazy; enumerate once. /// Samples; must be time-unique (the TVP has a PK on time) and non-empty when bound. /// Records for . - public static IEnumerable ToSampleRecords(IEnumerable rows) + public static IEnumerable ToSampleRecords(IEnumerable rows) { foreach (var r in rows) { - var c = ToColumns(r.Value); + var c = SampleColumnMapper.ToColumns(r.Value); var rec = new SqlDataRecord(SampleMeta); rec.SetDateTime(0, DateTime.SpecifyKind(r.TimeUtc, DateTimeKind.Utc)); rec.SetByte(1, c.ValueKind); if (c.NumValue is { } n) rec.SetDouble(2, n); else rec.SetDBNull(2); if (c.IntValue is { } i) rec.SetInt64(3, i); else rec.SetDBNull(3); if (c.StrValue is { } s) rec.SetString(4, s); else rec.SetDBNull(4); - rec.SetInt32(5, ToDbStatus(r.StatusCode)); + rec.SetInt32(5, SampleColumnMapper.ToDbStatus(r.StatusCode)); yield return rec; } } diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlAggregateQueries.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlAggregateQueries.cs index fa10b21b..050cb231 100644 --- a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlAggregateQueries.cs +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlAggregateQueries.cs @@ -1,22 +1,10 @@ -using ZB.MOM.WW.OtOpcUa.Core.Abstractions; +using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common; namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer; -/// The SQL shape that serves an aggregate. One query per family. -internal enum SqlAggregateFamily -{ - /// Average / Minimum / Maximum / Total / Count / Range / PercentGood / PercentBad: one GROUP BY. - Simple, - - /// Start / End / Delta: first + last usable sample per bucket (ROW_NUMBER). - StartEnd, - - /// MinimumActualTime / MaximumActualTime: extreme usable sample per bucket, ties ⇒ earliest. - ActualTime, -} - /// -/// Push-down T-SQL for processed (aggregate) reads (spec §4 "Processed"). Every query takes +/// Push-down T-SQL for processed (aggregate) reads (spec §4 "Processed"), one query per +/// ( picks it). Every query takes /// @id (int TagId), @start/@end (datetime2(7), the forward window /// [start, end)) and @intervalTicks (bigint bucket width in 100 ns ticks), and returns one row /// per non-empty bucket. The caller generates every interval and fills the empty ones. @@ -52,19 +40,6 @@ internal static class SqlAggregateQueries /// ValueKind of String (4) and DateTime (5): usable but not numeric. private const string NonNumericKinds = "4, 5"; - /// The family that serves , or null when this backend does not. - /// Requested aggregate. - /// The family, or null for an unsupported aggregate. - public static SqlAggregateFamily? FamilyOf(HistoryAggregateType aggregate) => aggregate switch - { - HistoryAggregateType.Average or HistoryAggregateType.Minimum or HistoryAggregateType.Maximum - or HistoryAggregateType.Total or HistoryAggregateType.Count or HistoryAggregateType.Range - or HistoryAggregateType.PercentGood or HistoryAggregateType.PercentBad => SqlAggregateFamily.Simple, - HistoryAggregateType.Start or HistoryAggregateType.End or HistoryAggregateType.Delta => SqlAggregateFamily.StartEnd, - HistoryAggregateType.MinimumActualTime or HistoryAggregateType.MaximumActualTime => SqlAggregateFamily.ActualTime, - _ => null, - }; - /// /// Average / Minimum / Maximum / Total / Count / Range / PercentGood / PercentBad. /// Family columns: 6 AvgValue, 7 MinValue, 8 MaxValue, 9 SumValue (float, NULL when no usable diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlHistorianStatusCodes.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlHistorianStatusCodes.cs deleted file mode 100644 index ea5b18e2..00000000 --- a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlHistorianStatusCodes.cs +++ /dev/null @@ -1,81 +0,0 @@ -namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer; - -/// -/// Every OPC UA status this backend emits, as named constants (inline literals are invisible to -/// StatusCodeParityTests). Values mirror Opc.Ua.StatusCodes. -/// -internal static class SqlHistorianStatusCodes -{ - public const uint Good = 0x00000000u; - public const uint GoodEntryInserted = 0x00A20000u; - public const uint GoodEntryReplaced = 0x00A30000u; - public const uint BadNoData = 0x809B0000u; - public const uint BadEntryExists = 0x809F0000u; - public const uint BadNoEntryExists = 0x80A00000u; - public const uint BadInvalidTimestamp = 0x80230000u; - public const uint BadAggregateInvalidInputs = 0x80D60000u; - public const uint UncertainDataSubNormal = 0x40A40000u; - - /// Stored for a non-finite Double (NaN / ±Infinity) that arrived with a non-Bad status. - public const uint BadOutOfRange = 0x803C0000u; -} - -/// -/// Bit masks over a 32-bit OPC UA status code (Part 4 §7.39). These are masks, not emitted codes, so their -/// names deliberately avoid the Good/Bad/Uncertain prefix the status-code parity scan keys on. -/// -internal static class StatusBits -{ - /// Severity + sub-code (top 16 bits) of a status, i.e. the code without info bits. - public const uint CodeBitsMask = 0xFFFF0000u; - - /// The two severity bits (Good = 00, Uncertain = 01, Bad = 10). - public const uint SeverityMask = 0xC0000000u; - - /// The severity bit set on every Bad code. - public const uint SeverityBadBit = 0x80000000u; - - /// The two historian info bits (Calculated, Interpolated) in the low bits of the info field. - public const uint HistorianBitsMask = 0x3u; - - /// True when has Bad severity. - /// A status code. - /// Whether the Bad severity bit is set. - public static bool IsBad(uint status) => (status & SeverityBadBit) != 0; - - /// True when has Good severity. - /// A status code. - /// Whether both severity bits are clear. - public static bool IsGood(uint status) => (status & SeverityMask) == 0; -} - -/// Part 4 §7.39 historian info bits OR-ed onto a status code. -internal static class HistorianInfoBits -{ - /// InfoType = DataValue. - public const uint InfoTypeDataValue = 0x00000400u; - - /// The value was produced by an aggregate calculation. - public const uint HistorianCalculated = 0x1u; - - /// The value was interpolated. - public const uint HistorianInterpolated = 0x2u; - - /// - /// Marks as a Calculated data value. Any historian bits the status already carries - /// (e.g. a stored Interpolated sample) are cleared first, so the result is never both. - /// - /// Base status code. - /// Status with InfoType and Calculated bits set. - public static uint WithCalculated(uint status) => - (status & ~StatusBits.HistorianBitsMask) | InfoTypeDataValue | HistorianCalculated; - - /// - /// Marks as an Interpolated data value. Any historian bits the status already - /// carries (e.g. a stored Calculated sample) are cleared first, so the result is never both. - /// - /// Base status code. - /// Status with InfoType and Interpolated bits set. - public static uint WithInterpolated(uint status) => - (status & ~StatusBits.HistorianBitsMask) | InfoTypeDataValue | HistorianInterpolated; -} diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerAlarmHistorianWriter.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerAlarmHistorianWriter.cs index a1b40406..715ff59c 100644 --- a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerAlarmHistorianWriter.cs +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerAlarmHistorianWriter.cs @@ -1,17 +1,16 @@ using System.Data; -using System.Security.Cryptography; -using System.Text; using Microsoft.Data.SqlClient; using Microsoft.Extensions.Logging; using Microsoft.Extensions.Logging.Abstractions; using ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian; using ZB.MOM.WW.OtOpcUa.Core.Abstractions; +using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common; namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer; /// /// over the SQL Event table. Per event: insert-if-absent keyed by a -/// deterministic , so the store-and-forward drain's at-least-once redelivery (and both +/// deterministic , so the store-and-forward drain's at-least-once redelivery (and both /// pair nodes writing the same event) converge on one row. Outcomes: Ack on insert or already-present; /// RetryPlease on transient SQL / schema-bootstrap / unclassified SQL errors and on cancellation; PermanentFail only /// for data errors (truncation, conversion, null, range, constraint); unknown/non-SQL exceptions retry. Never throws. @@ -76,12 +75,12 @@ internal sealed class SqlServerAlarmHistorianWriter : IAlarmHistorianWriter { await using var conn = await _conn.OpenAsync(ct).ConfigureAwait(false); await using var cmd = _conn.CreateCommand(conn, sql); - cmd.Parameters.Add(new SqlParameter("@id", SqlDbType.NVarChar, 64) { Value = ComputeEventId(e) }); + cmd.Parameters.Add(new SqlParameter("@id", SqlDbType.NVarChar, 64) { Value = AlarmEventIds.ComputeEventId(e) }); cmd.Parameters.Add(new SqlParameter("@src", SqlDbType.NVarChar, -1) { Value = e.EquipmentPath ?? "" }); cmd.Parameters.Add(new SqlParameter("@aid", SqlDbType.NVarChar, -1) { Value = e.AlarmId ?? "" }); cmd.Parameters.Add(new SqlParameter("@an", SqlDbType.NVarChar, -1) { Value = e.AlarmName ?? "" }); cmd.Parameters.Add(new SqlParameter("@at", SqlDbType.NVarChar, -1) { Value = e.AlarmTypeName ?? "" }); - cmd.Parameters.Add(new SqlParameter("@sev", SqlDbType.Int) { Value = ToNumericSeverity(e.Severity) }); + cmd.Parameters.Add(new SqlParameter("@sev", SqlDbType.Int) { Value = AlarmEventIds.ToNumericSeverity(e.Severity) }); cmd.Parameters.Add(new SqlParameter("@ek", SqlDbType.NVarChar, -1) { Value = e.EventKind ?? "" }); cmd.Parameters.Add(new SqlParameter("@msg", SqlDbType.NVarChar, -1) { Value = (object?)e.Message ?? DBNull.Value }); cmd.Parameters.Add(new SqlParameter("@usr", SqlDbType.NVarChar, -1) { Value = (object?)e.User ?? DBNull.Value }); @@ -116,50 +115,4 @@ internal sealed class SqlServerAlarmHistorianWriter : IAlarmHistorianWriter return HistorianWriteOutcome.PermanentFail; return HistorianWriteOutcome.RetryPlease; } - - // Same numeric Part 9 priorities the gateway path sends (AlarmEventMapper.ToNumericSeverity). - private static int ToNumericSeverity(AlarmSeverity s) => s switch - { - AlarmSeverity.Low => 200, - AlarmSeverity.Medium => 500, - AlarmSeverity.High => 700, - AlarmSeverity.Critical => 900, - _ => 500, - }; - - /// - /// Lowercase-hex SHA-256 over length-prefixed UTF-8 fields (little-endian int32 byte length, -1 for null): - /// EquipmentPath, AlarmId, EventKind, TimestampUtc.Ticks (decimal), User, Comment, Message. Length-prefixing keeps - /// field boundaries unambiguous ("a|b"+"c" ≠ "a"+"b|c"). - /// - /// The event. - /// 64 hex characters. - internal static string ComputeEventId(AlarmHistorianEvent e) - { - using var ms = new MemoryStream(); - Field(ms, e.EquipmentPath); - Field(ms, e.AlarmId); - Field(ms, e.EventKind); - Field(ms, e.TimestampUtc.Ticks.ToString(System.Globalization.CultureInfo.InvariantCulture)); - Field(ms, e.User); - Field(ms, e.Comment); - Field(ms, e.Message); - return Convert.ToHexStringLower(SHA256.HashData(ms.ToArray())); - } - - private static void Field(MemoryStream ms, string? value) - { - Span len = stackalloc byte[4]; - if (value is null) - { - System.Buffers.Binary.BinaryPrimitives.WriteInt32LittleEndian(len, -1); - ms.Write(len); - return; - } - - var bytes = Encoding.UTF8.GetBytes(value); - System.Buffers.Binary.BinaryPrimitives.WriteInt32LittleEndian(len, bytes.Length); - ms.Write(len); - ms.Write(bytes); - } } diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerHistorianDataSource.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerHistorianDataSource.cs index d2ef21b0..c580cdea 100644 --- a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerHistorianDataSource.cs +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerHistorianDataSource.cs @@ -4,6 +4,7 @@ using Microsoft.Data.SqlClient; using Microsoft.Extensions.Logging; using Microsoft.Extensions.Logging.Abstractions; using ZB.MOM.WW.OtOpcUa.Core.Abstractions; +using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common; namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer; @@ -39,7 +40,7 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer; /// on Raw and AtTime alike. /// /// -/// Long names. A tag name or event source name longer than +/// Long names. A tag name or event source name longer than /// can never be stored, so it is answered as not found (Raw: empty; AtTime / Processed: BadNoData; /// Events: none) without being truncated onto another series. /// @@ -47,7 +48,7 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer; public sealed class SqlServerHistorianDataSource : IHistorianDataSource { /// Spec §9.7: a processed read covering more intervals than this is refused. - public const int MaxProcessedBuckets = 100_000; + public const int MaxProcessedBuckets = BucketPlan.MaxBuckets; private const string SampleColumns = "SourceTimeUtc, ValueKind, NumValue, IntValue, StrValue, StatusCode, ServerTimeUtc"; @@ -57,14 +58,7 @@ public sealed class SqlServerHistorianDataSource : IHistorianDataSource private readonly ILogger _log; private readonly TimeProvider _time; - private readonly object _healthLock = new(); - private DateTime? _lastSuccessUtc; - private DateTime? _lastFailureUtc; - private string? _lastError; - private long _totalQueries; - private long _totalSuccesses; - private long _totalFailures; - private int _consecutiveFailures; + private readonly HistorianHealthCounters _health = new(); /// Creates the data source over the provider's shared connection, bootstrapper and tag catalog. /// Connection helper. @@ -192,7 +186,7 @@ public sealed class SqlServerHistorianDataSource : IHistorianDataSource StoredSample? p = reader.IsDBNull(1) ? null : ReadStored(reader, 1); StoredSample? n = reader.IsDBNull(8) ? null : ReadStored(reader, 8); var serverTime = p is { } ps && ps.TimeUtc == t ? SqlHistorianConnection.GetUtc(reader, 7) : now; - byTicks[t.Ticks] = AtTimeValue(t, p, n, serverTime, now); + byTicks[t.Ticks] = AtTimeEvaluator.Evaluate(t, p, n, serverTime, now); } } @@ -202,7 +196,7 @@ public sealed class SqlServerHistorianDataSource : IHistorianDataSource var t = DateTime.SpecifyKind(timestampsUtc[i], DateTimeKind.Utc); result[i] = byTicks.TryGetValue(t.Ticks, out var v) ? v - : new DataValueSnapshot(null, SqlHistorianStatusCodes.BadNoData, t, now); + : new DataValueSnapshot(null, HistorianStatusCodes.BadNoData, t, now); } RecordSuccess(); @@ -215,44 +209,6 @@ public sealed class SqlServerHistorianDataSource : IHistorianDataSource } } - /// Applies the AtTime rules (see ) to one requested time. - private static DataValueSnapshot AtTimeValue(DateTime t, StoredSample? prev, StoredSample? next, DateTime serverTime, DateTime now) - { - if (prev is not { } p) - return new DataValueSnapshot(null, SqlHistorianStatusCodes.BadNoData, t, serverTime); - if (p.TimeUtc == t) - return new DataValueSnapshot(p.Value.ToObject(), p.Status, t, serverTime); - - // Inside an outage: report the outage's own Bad code, never an upgraded Uncertain. - if (!p.IsUsable) - return new DataValueSnapshot(null, p.Status & StatusBits.CodeBitsMask, t, serverTime); - - // Usable previous, Bad next: hold the previous value up to the outage; do not interpolate into it. - if (next is { IsUsable: false }) - return new DataValueSnapshot(p.Value.ToObject(), - HistorianInfoBits.WithInterpolated(SqlHistorianStatusCodes.UncertainDataSubNormal), t, serverTime); - - if (next is { } n && p.Value.Kind is HistorianValueKind.Double or HistorianValueKind.Int64 - && n.Value.TryGetDouble(out var nv)) - { - var fraction = (double)(t.Ticks - p.TimeUtc.Ticks) / (n.TimeUtc.Ticks - p.TimeUtc.Ticks); - var v = p.Value.Number + (nv - p.Value.Number) * fraction; - object value = p.Value.Kind == HistorianValueKind.Int64 - ? (long)Math.Round(v, MidpointRounding.AwayFromZero) - : v; - var status = p.IsGood && n.IsGood ? SqlHistorianStatusCodes.Good : SqlHistorianStatusCodes.UncertainDataSubNormal; - return new DataValueSnapshot(value, HistorianInfoBits.WithInterpolated(status), t, serverTime); - } - - // Beyond the last sample AND beyond now: the value is held into the future, so it is not known to be Good. - if (next is null && t > now) - return new DataValueSnapshot(p.Value.ToObject(), - HistorianInfoBits.WithInterpolated(SqlHistorianStatusCodes.UncertainDataSubNormal), t, serverTime); - - // Stepped (non-numeric kind, or after the last sample): hold the previous value and its status. - return new DataValueSnapshot(p.Value.ToObject(), HistorianInfoBits.WithInterpolated(p.Status), t, serverTime); - } - // ------------------------------------------------------------------------------------------ Events /// @@ -269,7 +225,7 @@ public sealed class SqlServerHistorianDataSource : IHistorianDataSource { // No stored event can carry a source longer than the column (the alarm writer dead-letters it rather than // truncating), so an over-long filter matches nothing — never a truncated prefix of someone else's source. - if (sourceName is not null && sourceName.Length > SqlServerTagCatalog.MaxNameLength) + if (sourceName is not null && sourceName.Length > HistorianTagNames.MaxLength) return new HistoricalEventsResult([], null); try { @@ -301,10 +257,10 @@ public sealed class SqlServerHistorianDataSource : IHistorianDataSource SqlHistorianConnection.GetUtc(reader, 2), SqlHistorianConnection.GetUtc(reader, 3), reader.IsDBNull(4) ? null : reader.GetString(4), - (ushort)Math.Clamp(reader.GetInt32(5), 0, ushort.MaxValue)); + HistoricalEventRows.ClampSeverity(reader.GetInt32(5))); if (bounded && events.Count == maxEvents) { - continuation = Encoding.UTF8.GetBytes($"{e.EventTimeUtc.Ticks}|{e.EventId}"); + continuation = Encoding.UTF8.GetBytes(EventContinuation.Encode(e.EventTimeUtc, e.EventId)); break; } events.Add(e); @@ -352,57 +308,38 @@ public sealed class SqlServerHistorianDataSource : IHistorianDataSource HistoryAggregateType aggregate, CancellationToken cancellationToken) { ArgumentNullException.ThrowIfNull(fullReference); - var family = SqlAggregateQueries.FamilyOf(aggregate) ?? throw new HistoryAggregateNotSupportedException(aggregate); - - var reverse = endUtc < startUtc; - var (start, end) = reverse ? (endUtc, startUtc) : (startUtc, endUtc); - var windowTicks = end.Ticks - start.Ticks; - // Everything stays in 100 ns ticks (≤ ≈3.2e18 across 0001–9999, inside a long) — no ns scaling that could - // wrap. An interval wider than the window is one bucket, so clamp it to the window; the ceiling division - // is written without "+ interval - 1" so a TimeSpan near MaxValue cannot overflow either. - var intervalTicks = interval > TimeSpan.Zero ? Math.Min(interval.Ticks, windowTicks) : windowTicks; - long bucketCount = windowTicks == 0 ? 0 : windowTicks / intervalTicks + (windowTicks % intervalTicks == 0 ? 0 : 1); - if (bucketCount > MaxProcessedBuckets) - throw new ArgumentOutOfRangeException(nameof(interval), bucketCount, - $"A processed read may cover at most {MaxProcessedBuckets} intervals; this window holds {bucketCount}."); + var family = AggregateFamilies.FamilyOf(aggregate) ?? throw new HistoryAggregateNotSupportedException(aggregate); + var plan = BucketPlan.Create(startUtc, endUtc, interval); try { var now = _time.GetUtcNow().UtcDateTime; - var rows = new Dictionary(); - var tagId = bucketCount == 0 ? null : await ResolveAsync(fullReference, cancellationToken).ConfigureAwait(false); + var evaluated = new Dictionary(); + var tagId = plan.BucketCount == 0 ? null : await ResolveAsync(fullReference, cancellationToken).ConfigureAwait(false); if (tagId is { } id) { var sample = _conn.Q("Sample"); var sql = family switch { - SqlAggregateFamily.Simple => SqlAggregateQueries.Simple(sample), - SqlAggregateFamily.StartEnd => SqlAggregateQueries.StartEnd(sample), + AggregateFamily.Simple => SqlAggregateQueries.Simple(sample), + AggregateFamily.StartEnd => SqlAggregateQueries.StartEnd(sample), _ => SqlAggregateQueries.ActualTime(sample, aggregate == HistoryAggregateType.MaximumActualTime), }; await using var conn = await _conn.OpenAsync(cancellationToken).ConfigureAwait(false); await using var cmd = _conn.CreateCommand(conn, sql); cmd.Parameters.Add(new SqlParameter("@id", SqlDbType.Int) { Value = id }); - SqlHistorianConnection.AddDateTime2(cmd, "@start", DateTime.SpecifyKind(start, DateTimeKind.Utc)); - SqlHistorianConnection.AddDateTime2(cmd, "@end", DateTime.SpecifyKind(end, DateTimeKind.Utc)); - cmd.Parameters.Add(new SqlParameter("@intervalTicks", SqlDbType.BigInt) { Value = intervalTicks }); + SqlHistorianConnection.AddDateTime2(cmd, "@start", DateTime.SpecifyKind(plan.StartUtc, DateTimeKind.Utc)); + SqlHistorianConnection.AddDateTime2(cmd, "@end", DateTime.SpecifyKind(plan.EndUtc, DateTimeKind.Utc)); + cmd.Parameters.Add(new SqlParameter("@intervalTicks", SqlDbType.BigInt) { Value = plan.IntervalTicks }); await using var reader = await cmd.ExecuteReaderAsync(cancellationToken).ConfigureAwait(false); while (await reader.ReadAsync(cancellationToken).ConfigureAwait(false)) { var row = ReadBucket(reader, family); - rows[row.Bucket] = row; + evaluated[row.Bucket] = AggregateEvaluator.Evaluate(aggregate, row, plan.BucketStart(row.Bucket), now); } } - var result = new List((int)bucketCount); - for (long i = 0; i < bucketCount; i++) - { - var bucketStart = DateTime.SpecifyKind(new DateTime(start.Ticks + i * intervalTicks), DateTimeKind.Utc); - result.Add(rows.TryGetValue(i, out var row) - ? Evaluate(aggregate, row, bucketStart, now) - : NoData(bucketStart, now)); - } - if (reverse) result.Reverse(); + var result = plan.Assemble(evaluated, bucketStart => AggregateEvaluator.NoData(bucketStart, now)); RecordSuccess(); return new HistoryReadResult(result, ContinuationPoint: null); @@ -414,56 +351,13 @@ public sealed class SqlServerHistorianDataSource : IHistorianDataSource } } - private static DataValueSnapshot NoData(DateTime t, DateTime now) => - new(null, HistorianInfoBits.WithCalculated(SqlHistorianStatusCodes.BadNoData), t, now); - - private static DataValueSnapshot Evaluate(HistoryAggregateType aggregate, BucketRow r, DateTime bucketStart, DateTime now) - { - DataValueSnapshot Calc(object? value, uint status, DateTime? t = null) => - new(value, HistorianInfoBits.WithCalculated(status), t ?? bucketStart, now); - - if (r.Total == 0) return NoData(bucketStart, now); - switch (aggregate) - { - case HistoryAggregateType.PercentGood: return Calc(100d * r.Good / r.Total, SqlHistorianStatusCodes.Good); - case HistoryAggregateType.PercentBad: return Calc(100d * r.Bad / r.Total, SqlHistorianStatusCodes.Good); - } - if (r.Usable == 0) return NoData(bucketStart, now); - - var anyText = aggregate is HistoryAggregateType.Count or HistoryAggregateType.Start or HistoryAggregateType.End; - if (!anyText && r.NonNumeric > 0) - return Calc(null, SqlHistorianStatusCodes.BadAggregateInvalidInputs); - - var status = r.Good == r.Total ? SqlHistorianStatusCodes.Good : SqlHistorianStatusCodes.UncertainDataSubNormal; - switch (aggregate) - { - case HistoryAggregateType.Count: return Calc(r.Usable, status); - case HistoryAggregateType.Average: return Calc(r.Avg, status); - case HistoryAggregateType.Minimum: return Calc(r.Min, status); - case HistoryAggregateType.Maximum: return Calc(r.Max, status); - case HistoryAggregateType.Total: return Calc(r.Sum, status); - case HistoryAggregateType.Range: return Calc(r.Max - r.Min, status); - case HistoryAggregateType.Start: return Calc(r.First.ToObject(), status); - case HistoryAggregateType.End: return Calc(r.Last.ToObject(), status); - case HistoryAggregateType.Delta: - return r.First.TryGetDouble(out var a) && r.Last.TryGetDouble(out var b) - ? Calc(b - a, status) - : Calc(null, SqlHistorianStatusCodes.BadAggregateInvalidInputs); - case HistoryAggregateType.MinimumActualTime: - case HistoryAggregateType.MaximumActualTime: - return Calc(r.Extreme, status, r.ExtremeTime); - default: - throw new HistoryAggregateNotSupportedException(aggregate); - } - } - - private static BucketRow ReadBucket(SqlDataReader reader, SqlAggregateFamily family) + private static BucketRow ReadBucket(SqlDataReader reader, AggregateFamily family) { static double? D(SqlDataReader r, int i) => r.IsDBNull(i) ? null : r.GetDouble(i); static long? L(SqlDataReader r, int i) => r.IsDBNull(i) ? null : r.GetInt64(i); static string? S(SqlDataReader r, int i) => r.IsDBNull(i) ? null : r.GetString(i); static HistorianValue V(SqlDataReader r, int i) => - r.IsDBNull(i) ? HistorianValue.Empty : SampleRowMapper.FromColumns(r.GetByte(i), D(r, i + 1), L(r, i + 2), S(r, i + 3)); + r.IsDBNull(i) ? HistorianValue.Empty : SampleColumnMapper.FromColumns(r.GetByte(i), D(r, i + 1), L(r, i + 2), S(r, i + 3)); var row = new BucketRow { @@ -477,13 +371,13 @@ public sealed class SqlServerHistorianDataSource : IHistorianDataSource var f = SqlAggregateQueries.FirstFamilyColumn; switch (family) { - case SqlAggregateFamily.Simple: + case AggregateFamily.Simple: row.Avg = D(reader, f); row.Min = D(reader, f + 1); row.Max = D(reader, f + 2); row.Sum = D(reader, f + 3); break; - case SqlAggregateFamily.StartEnd: + case AggregateFamily.StartEnd: row.First = V(reader, f); row.Last = V(reader, f + 5); break; @@ -495,44 +389,23 @@ public sealed class SqlServerHistorianDataSource : IHistorianDataSource return row; } - /// One aggregate query row (one non-empty bucket). - private sealed class BucketRow - { - public long Bucket; - public int Total, Usable, Good, Bad, NonNumeric; - public double? Avg, Min, Max, Sum, Extreme; - public DateTime? ExtremeTime; - public HistorianValue First, Last; - } - // ------------------------------------------------------------------------------------------ Shared - /// A stored sample read from the {SampleColumns} column run starting at an ordinal. - private readonly record struct StoredSample(DateTime TimeUtc, HistorianValue Value, uint Status) - { - /// Severity Good or Uncertain, and a value present (an Empty sample is an outage marker). - public bool IsUsable => !StatusBits.IsBad(Status) && Value.Kind != HistorianValueKind.Empty; - - /// Usable with severity Good. - public bool IsGood => IsUsable && StatusBits.IsGood(Status); - } - /// /// Reads SourceTimeUtc, ValueKind, NumValue, IntValue, StrValue, StatusCode starting at . - /// An Empty (no-value) row whose stored status is not Bad is clamped to BadNoData — defence in depth for - /// rows written before the writer forced every Empty sample Bad. + /// An Empty (no-value) row whose stored status is not Bad is clamped to BadNoData + /// () — defence in depth for rows written before the writer forced every + /// Empty sample Bad. /// private static StoredSample ReadStored(SqlDataReader r, int o) { - var value = SampleRowMapper.FromColumns( + var value = SampleColumnMapper.FromColumns( r.GetByte(o + 1), r.IsDBNull(o + 2) ? null : r.GetDouble(o + 2), r.IsDBNull(o + 3) ? null : r.GetInt64(o + 3), r.IsDBNull(o + 4) ? null : r.GetString(o + 4)); - var status = SampleRowMapper.FromDbStatus(r.GetInt32(o + 5)); - if (value.Kind == HistorianValueKind.Empty && !StatusBits.IsBad(status)) - status = SqlHistorianStatusCodes.BadNoData; - return new StoredSample(SqlHistorianConnection.GetUtc(r, o), value, status); + var status = SampleColumnMapper.FromDbStatus(r.GetInt32(o + 5)); + return StoredSample.FromStored(SqlHistorianConnection.GetUtc(r, o), value, status); } /// Read-only tag lookup (never creates a row); runs the schema bootstrap on a cache miss. @@ -541,50 +414,15 @@ public sealed class SqlServerHistorianDataSource : IHistorianDataSource // ------------------------------------------------------------------------------------------ Health /// - public HistorianHealthSnapshot GetHealthSnapshot() - { - lock (_healthLock) - { - var open = _totalSuccesses > 0 && _consecutiveFailures == 0; - return new HistorianHealthSnapshot( - TotalQueries: _totalQueries, - TotalSuccesses: _totalSuccesses, - TotalFailures: _totalFailures, - ConsecutiveFailures: _consecutiveFailures, - LastSuccessTime: _lastSuccessUtc, - LastFailureTime: _lastFailureUtc, - LastError: _lastError, - ProcessConnectionOpen: open, - EventConnectionOpen: open, - ActiveProcessNode: null, - ActiveEventNode: null, - Nodes: []); - } - } + public HistorianHealthSnapshot GetHealthSnapshot() => _health.Snapshot(); - private void RecordSuccess() - { - lock (_healthLock) - { - _totalQueries++; - _totalSuccesses++; - _consecutiveFailures = 0; - _lastSuccessUtc = DateTime.UtcNow; - } - } + private void RecordSuccess() => _health.RecordSuccess(); private void RecordFailure(Exception ex) { - // A cancelled read (deadline / client abort) is not a backend fault. + // A cancelled read (deadline / client abort) is not a backend fault (the counters ignore it too). if (ex is OperationCanceledException) return; - lock (_healthLock) - { - _totalQueries++; - _totalFailures++; - _consecutiveFailures++; - _lastFailureUtc = DateTime.UtcNow; - _lastError = $"{ex.GetType().Name}: {ex.Message}"; - } + _health.RecordFailure(ex); _log.LogDebug("SQL historian read failed ({ExceptionType})", ex.GetType().Name); } diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerHistorianDataUpdater.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerHistorianDataUpdater.cs index efd0a65d..057dfe45 100644 --- a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerHistorianDataUpdater.cs +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerHistorianDataUpdater.cs @@ -1,18 +1,9 @@ using Microsoft.Data.SqlClient; using ZB.MOM.WW.OtOpcUa.Core.Abstractions; +using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common; namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer; -/// Status codes owned by the updater (named consts so StatusCodeParityTests sees them). -internal static class SqlHistorianUpdateStatusCodes -{ - /// - /// A value that cannot be stored: a Double NaN / ±Infinity, an Empty (null) value with a non-Bad status, or a - /// tag name longer than . - /// - public const uint BadInvalidArgument = 0x80AB0000u; -} - /// /// OPC UA HistoryUpdate against the SQL Server sample table (spec §4 "Updates"). One transaction per call; /// exactly one status per input item, in input order. @@ -26,7 +17,7 @@ internal static class SqlHistorianUpdateStatusCodes /// accepted with a Bad status (an operator-authored outage marker); with a Good or Uncertain status it is /// BadInvalidArgument, because a stored sample is never served better than "no value". Other items /// still proceed. -/// Tag names over characters are refused, never +/// Tag names over characters are refused, never /// truncated: every item of the call (and DeleteRaw's single result) reports BadInvalidArgument and nothing /// is written. /// Unknown tag. Insert/Update create it via ; Replace and the @@ -52,70 +43,26 @@ internal sealed class SqlServerHistorianDataUpdater : IHistorianDataUpdater { ArgumentNullException.ThrowIfNull(tag); ArgumentNullException.ThrowIfNull(values); - var results = new uint[values.Count]; - if (!SqlServerTagCatalog.IsValidName(tag)) - { - Array.Fill(results, SqlHistorianUpdateStatusCodes.BadInvalidArgument); - return results; - } - var winner = new Dictionary(); // tick -> index of the item actually sent - for (var i = 0; i < values.Count; i++) - { - var v = values[i]; - if (v.TimestampUtc is not { } ts || ts == DateTime.MinValue) - { - results[i] = SqlHistorianStatusCodes.BadInvalidTimestamp; - continue; - } - if ((v.Value.Kind == HistorianValueKind.Double && !double.IsFinite(v.Value.Number)) - || (v.Value.Kind == HistorianValueKind.Empty && !StatusBits.IsBad(v.StatusCode))) - { - results[i] = SqlHistorianUpdateStatusCodes.BadInvalidArgument; - continue; - } - if (!winner.TryGetValue(ts.Ticks, out _)) - winner[ts.Ticks] = i; - else if (mode == HistoryUpdateMode.Insert) - results[i] = SqlHistorianStatusCodes.BadEntryExists; // first occurrence stays the winner - else - winner[ts.Ticks] = i; // last wins - } - - var send = winner.Values.OrderBy(i => i).ToList(); - if (send.Count == 0) return results; + var plan = HistoryUpdatePlanner.Plan(tag, mode, values); + if (plan.Winners.Count == 0) return HistoryUpdatePlanner.Complete(plan, mode, new Dictionary()); var tagId = mode == HistoryUpdateMode.Replace ? await _tags.TryGetTagIdAsync(tag, ct).ConfigureAwait(false) - : await _tags.GetOrCreateTagIdAsync(tag, SqlHistorianDataTypes.FromKind(values[send[0]].Value.Kind), ct).ConfigureAwait(false); + : await _tags.GetOrCreateTagIdAsync(tag, HistorianDataTypes.FromKind(plan.Winners[0].Row.Value.Kind), ct).ConfigureAwait(false); var outcome = tagId is null - ? new Dictionary() - : await MergeAsync(tagId.Value, mode, send.Select(i => new SqlSampleRow( - values[i].TimestampUtc!.Value, values[i].Value, values[i].StatusCode)).ToList(), ct).ConfigureAwait(false); - - // No OUTPUT row means the MERGE's absent branch: Insert => matched (exists); Replace => not matched. - var absent = mode == HistoryUpdateMode.Replace - ? SqlHistorianStatusCodes.BadNoEntryExists - : SqlHistorianStatusCodes.BadEntryExists; - foreach (var i in send) - results[i] = outcome.TryGetValue(values[i].TimestampUtc!.Value.Ticks, out var code) ? code : absent; - - // Superseded earlier duplicates (Replace/Update) share the winner's code. - if (mode != HistoryUpdateMode.Insert) - for (var i = 0; i < values.Count; i++) - if (values[i].TimestampUtc is { } ts && results[i] == 0 - && winner.TryGetValue(ts.Ticks, out var w) && w != i) - results[i] = results[w]; - return results; + ? new Dictionary() + : await MergeAsync(tagId.Value, mode, plan.Winners.Select(w => w.Row).ToList(), ct).ConfigureAwait(false); + return HistoryUpdatePlanner.Complete(plan, mode, outcome); } /// public async Task DeleteRawAsync(string tag, DateTime startUtc, DateTime endUtc, CancellationToken ct) { ArgumentNullException.ThrowIfNull(tag); - if (!SqlServerTagCatalog.IsValidName(tag)) return SqlHistorianUpdateStatusCodes.BadInvalidArgument; + if (!HistorianTagNames.IsValid(tag)) return HistorianStatusCodes.BadInvalidArgument; var tagId = await _tags.TryGetTagIdAsync(tag, ct).ConfigureAwait(false); - if (tagId is null) return SqlHistorianStatusCodes.BadNoData; + if (tagId is null) return HistoryUpdatePlanner.DeleteRawResult(tagKnown: false, 0); await using var conn = await _conn.OpenAsync(ct).ConfigureAwait(false); await using var tx = (SqlTransaction)await conn.BeginTransactionAsync(ct).ConfigureAwait(false); int rows; @@ -128,7 +75,7 @@ internal sealed class SqlServerHistorianDataUpdater : IHistorianDataUpdater rows = await cmd.ExecuteNonQueryAsync(ct).ConfigureAwait(false); } await tx.CommitAsync(ct).ConfigureAwait(false); - return rows > 0 ? SqlHistorianStatusCodes.Good : SqlHistorianStatusCodes.BadNoData; + return HistoryUpdatePlanner.DeleteRawResult(tagKnown: true, rows); } /// @@ -136,22 +83,19 @@ internal sealed class SqlServerHistorianDataUpdater : IHistorianDataUpdater { ArgumentNullException.ThrowIfNull(tag); ArgumentNullException.ThrowIfNull(timesUtc); - var results = new uint[timesUtc.Count]; - if (!SqlServerTagCatalog.IsValidName(tag)) + if (!HistorianTagNames.IsValid(tag)) { - Array.Fill(results, SqlHistorianUpdateStatusCodes.BadInvalidArgument); - return results; + var refused = new uint[timesUtc.Count]; + Array.Fill(refused, HistorianStatusCodes.BadInvalidArgument); + return refused; } - var first = new Dictionary(); - for (var i = 0; i < timesUtc.Count; i++) - { - results[i] = SqlHistorianStatusCodes.BadNoEntryExists; - if (timesUtc[i] == DateTime.MinValue) results[i] = SqlHistorianStatusCodes.BadInvalidTimestamp; - else first.TryAdd(timesUtc[i].Ticks, i); - } - if (first.Count == 0) return results; + var distinct = new HashSet(); + foreach (var t in timesUtc) + if (t != DateTime.MinValue) distinct.Add(t.Ticks); + var deleted = new HashSet(); + if (distinct.Count == 0) return HistoryUpdatePlanner.DeleteAtTimeResults(timesUtc, deleted, tagKnown: false); var tagId = await _tags.TryGetTagIdAsync(tag, ct).ConfigureAwait(false); - if (tagId is null) return results; + if (tagId is null) return HistoryUpdatePlanner.DeleteAtTimeResults(timesUtc, deleted, tagKnown: false); await using var conn = await _conn.OpenAsync(ct).ConfigureAwait(false); await using var tx = (SqlTransaction)await conn.BeginTransactionAsync(ct).ConfigureAwait(false); @@ -160,18 +104,17 @@ internal sealed class SqlServerHistorianDataUpdater : IHistorianDataUpdater { cmd.Parameters.Add(new SqlParameter("@tagId", System.Data.SqlDbType.Int) { Value = tagId.Value }); _conn.AddStructured(cmd, "@times", "TimeRow", - SampleRowMapper.ToTimeRecords(first.Keys.Select(t => new DateTime(t, DateTimeKind.Utc)))); + SampleRowMapper.ToTimeRecords(distinct.Select(t => new DateTime(t, DateTimeKind.Utc)))); await using var rd = await cmd.ExecuteReaderAsync(ct).ConfigureAwait(false); while (await rd.ReadAsync(ct).ConfigureAwait(false)) - if (first.TryGetValue(SqlHistorianConnection.GetUtc(rd, 0).Ticks, out var idx)) - results[idx] = SqlHistorianStatusCodes.Good; + deleted.Add(SqlHistorianConnection.GetUtc(rd, 0).Ticks); } await tx.CommitAsync(ct).ConfigureAwait(false); - return results; + return HistoryUpdatePlanner.DeleteAtTimeResults(timesUtc, deleted, tagKnown: true); } - private async Task> MergeAsync(int tagId, HistoryUpdateMode mode, - List rows, CancellationToken ct) + private async Task> MergeAsync(int tagId, HistoryUpdateMode mode, + List rows, CancellationToken ct) { var sql = $"MERGE {_conn.Q("Sample")} WITH (HOLDLOCK) AS t USING @rows AS r ON t.TagId = @tagId AND t.SourceTimeUtc = r.SourceTimeUtc "; if (mode != HistoryUpdateMode.Insert) @@ -182,7 +125,7 @@ internal sealed class SqlServerHistorianDataUpdater : IHistorianDataUpdater "VALUES (@tagId, r.SourceTimeUtc, r.ValueKind, r.NumValue, r.IntValue, r.StrValue, r.StatusCode, SYSUTCDATETIME()) "; sql += "OUTPUT $action, r.SourceTimeUtc;"; - var outcome = new Dictionary(); + var outcome = new Dictionary(); await using var conn = await _conn.OpenAsync(ct).ConfigureAwait(false); await using var tx = (SqlTransaction)await conn.BeginTransactionAsync(ct).ConfigureAwait(false); await using (var cmd = _conn.CreateCommand(conn, sql, tx)) @@ -192,8 +135,8 @@ internal sealed class SqlServerHistorianDataUpdater : IHistorianDataUpdater await using var rd = await cmd.ExecuteReaderAsync(ct).ConfigureAwait(false); while (await rd.ReadAsync(ct).ConfigureAwait(false)) outcome[SqlHistorianConnection.GetUtc(rd, 1).Ticks] = rd.GetString(0) == "INSERT" - ? SqlHistorianStatusCodes.GoodEntryInserted - : SqlHistorianStatusCodes.GoodEntryReplaced; + ? UpsertOutcome.Inserted + : UpsertOutcome.Replaced; } await tx.CommitAsync(ct).ConfigureAwait(false); return outcome; diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerHistorianOptions.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerHistorianOptions.cs index ed23c993..d5669b05 100644 --- a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerHistorianOptions.cs +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerHistorianOptions.cs @@ -44,11 +44,3 @@ public sealed partial class SqlServerHistorianOptions return errors; } } - -/// Thrown when AutoCreateSchema=false and the historian schema is absent or behind. -public sealed class HistorianSchemaMissingException : Exception -{ - /// Creates the exception. - /// Message naming the missing version. - public HistorianSchemaMissingException(string message) : base(message) { } -} diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerHistorianValueWriter.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerHistorianValueWriter.cs index 75f10757..4efa8da6 100644 --- a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerHistorianValueWriter.cs +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerHistorianValueWriter.cs @@ -2,6 +2,7 @@ using Microsoft.Data.SqlClient; using Microsoft.Extensions.Logging; using Microsoft.Extensions.Logging.Abstractions; using ZB.MOM.WW.OtOpcUa.Core.Abstractions; +using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common; namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer; @@ -21,6 +22,7 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer; /// Tag names over characters are refused (the write /// returns false, a per-tag failure for the recorder's #538 quarantine), never truncated onto another tag. /// Duplicate timestamps within a batch collapse to the first (matching insert-if-absent across batches). +/// The batch normalisation itself is . /// internal sealed class SqlServerHistorianValueWriter : IHistorianValueWriter { @@ -51,7 +53,7 @@ internal sealed class SqlServerHistorianValueWriter : IHistorianValueWriter { try { - if (!SqlServerTagCatalog.IsValidName(tag)) + if (!HistorianTagNames.IsValid(tag)) { _log.LogWarning("SQL historian live-value write refused: tag name is missing or longer than {Max} characters.", SqlServerTagCatalog.MaxNameLength); @@ -60,8 +62,8 @@ internal sealed class SqlServerHistorianValueWriter : IHistorianValueWriter await _bootstrap.EnsureAsync(ct).ConfigureAwait(false); if (values is null || values.Count == 0) return true; - var rows = Prepare(values, _utcNow); - var dataType = SqlHistorianDataTypes.FromKind(rows.Select(r => r.Value.Kind).FirstOrDefault(k => k != HistorianValueKind.Empty)); + var rows = LiveWritePreparer.Prepare(values, _utcNow); + var dataType = HistorianDataTypes.FromKind(rows.Select(r => r.Value.Kind).FirstOrDefault(k => k != HistorianValueKind.Empty)); var tagId = await _catalog.GetOrCreateTagIdAsync(tag, dataType, ct).ConfigureAwait(false); var sql = @@ -94,38 +96,4 @@ internal sealed class SqlServerHistorianValueWriter : IHistorianValueWriter return false; } } - - /// - /// Normalises a batch for the TVP: null timestamp ⇒ now, non-finite doubles ⇒ Empty, every Empty forced Bad - /// (non-finite ⇒ BadOutOfRange, otherwise BadNoData, an existing Bad code kept), and duplicate - /// timestamps collapsed to the first. - /// - /// Input samples, in append order. - /// Clock for null timestamps. - /// Time-unique rows. - internal static List Prepare(IReadOnlyList values, Func utcNow) - { - var seen = new HashSet(); - var rows = new List(values.Count); - foreach (var v in values) - { - var t = v.TimestampUtc ?? utcNow(); - t = t.Kind == DateTimeKind.Local ? t.ToUniversalTime() : DateTime.SpecifyKind(t, DateTimeKind.Utc); - if (!seen.Add(t.Ticks)) continue; - var value = v.Value; - var status = v.StatusCode; - if (value.Kind == HistorianValueKind.Double && !double.IsFinite(value.Number)) - { - value = HistorianValue.Empty; - if (!StatusBits.IsBad(status)) status = SqlHistorianStatusCodes.BadOutOfRange; - } - else if (value.Kind == HistorianValueKind.Empty && !StatusBits.IsBad(status)) - { - status = SqlHistorianStatusCodes.BadNoData; - } - rows.Add(new SqlSampleRow(t, value, status)); - } - - return rows; - } } diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerSchemaBootstrapper.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerSchemaBootstrapper.cs index 528b48fd..5539e251 100644 --- a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerSchemaBootstrapper.cs +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerSchemaBootstrapper.cs @@ -1,17 +1,17 @@ using System.Data; using System.Reflection; -using System.Runtime.ExceptionServices; using System.Text.RegularExpressions; using Microsoft.Data.SqlClient; using Microsoft.Extensions.Logging; using Microsoft.Extensions.Logging.Abstractions; +using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common; namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer; /// /// Lazily applies the embedded, idempotent schema scripts once per instance, under a SQL application lock /// so concurrent nodes/instances serialise. A failure is NOT latched: an after -/// retries. Within that window the last failure is re-thrown without touching +/// retries (). Within that window the last failure is re-thrown without touching /// SQL, so a missing schema (or a SQL outage) does not serialise every write, read and alarm event behind one /// semaphore and one round-trip each. /// With AutoCreateSchema=false it only verifies MAX(Version) >= 1. @@ -24,17 +24,12 @@ internal sealed partial class SqlServerSchemaBootstrapper private const string LockResource = "OtOpcUaHistorianSchema"; private const int LockTimeoutMs = 30_000; - /// How long the last bootstrap failure is re-thrown before SQL is tried again. - public static readonly TimeSpan FailureCacheDuration = TimeSpan.FromSeconds(5); - private readonly SqlHistorianConnection _conn; private readonly bool _autoCreate; private readonly ILogger _log; - private readonly TimeProvider _time; private readonly SemaphoreSlim _gate = new(1, 1); + private readonly BootstrapFailureCache _failure; // guarded by _gate private volatile bool _ready; - private ExceptionDispatchInfo? _lastFailure; // guarded by _gate - private DateTimeOffset _retryAfter; // guarded by _gate /// Creates the bootstrapper. /// Connection helper. @@ -47,7 +42,7 @@ internal sealed partial class SqlServerSchemaBootstrapper _conn = conn; _autoCreate = options.AutoCreateSchema; _log = log ?? NullLogger.Instance; - _time = timeProvider ?? TimeProvider.System; + _failure = new BootstrapFailureCache(timeProvider ?? TimeProvider.System, BootstrapFailureCache.DefaultDuration); } /// Ensures the schema is ready; no-op after the first success. @@ -61,23 +56,22 @@ internal sealed partial class SqlServerSchemaBootstrapper try { if (_ready) return; - if (_lastFailure is { } cached && _time.GetUtcNow() < _retryAfter) cached.Throw(); + _failure.ThrowIfCached(); try { await using var conn = await _conn.OpenAsync(ct).ConfigureAwait(false); if (_autoCreate) await ApplyAsync(conn, ct).ConfigureAwait(false); else await VerifyAsync(conn, ct).ConfigureAwait(false); _ready = true; - _lastFailure = null; + _failure.Clear(); } catch (Exception ex) when (ex is not OperationCanceledException || !ct.IsCancellationRequested) { // A caller's own cancellation is not a backend failure and is not cached. - _lastFailure = ExceptionDispatchInfo.Capture(ex); - _retryAfter = _time.GetUtcNow() + FailureCacheDuration; + _failure.Record(ex); _log.LogWarning("Historian schema bootstrap for [{Schema}] failed ({Exception}: {Message}); " + "historian calls fail fast for {Seconds} s before retrying.", - _conn.Schema, ex.GetType().Name, ex.Message, FailureCacheDuration.TotalSeconds); + _conn.Schema, ex.GetType().Name, ex.Message, BootstrapFailureCache.DefaultDuration.TotalSeconds); throw; } } diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerTagCatalog.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerTagCatalog.cs index b3e16121..1e5071e6 100644 --- a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerTagCatalog.cs +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerTagCatalog.cs @@ -1,5 +1,6 @@ using System.Collections.Concurrent; using Microsoft.Data.SqlClient; +using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common; namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer; @@ -13,8 +14,8 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer; /// internal sealed class SqlServerTagCatalog { - /// Width of the Tag.Name column; longer names are refused. - public const int MaxNameLength = 400; + /// Width of the Tag.Name column (= ); longer names are refused. + public const int MaxNameLength = HistorianTagNames.MaxLength; private readonly SqlHistorianConnection _conn; private readonly SqlServerSchemaBootstrapper _bootstrap; @@ -29,18 +30,13 @@ internal sealed class SqlServerTagCatalog _bootstrap = bootstrap; } - /// True when fits the Tag.Name column without truncation. - /// Historian tag name. - /// Whether the name is non-null and at most characters. - public static bool IsValidName(string? name) => name is not null && name.Length <= MaxNameLength; - /// Read-only lookup; never creates a row. A name over is not found. /// Historian tag name. /// Cancellation. /// The TagId, or null when the tag does not exist. public async Task TryGetTagIdAsync(string name, CancellationToken ct) { - if (!IsValidName(name)) return null; + if (!HistorianTagNames.IsValid(name)) return null; if (_cache.TryGetValue(name, out var cached)) return cached; await _bootstrap.EnsureAsync(ct).ConfigureAwait(false); await using var conn = await _conn.OpenAsync(ct).ConfigureAwait(false); @@ -57,7 +53,7 @@ internal sealed class SqlServerTagCatalog /// The name is longer than . public async Task GetOrCreateTagIdAsync(string name, byte dataType, CancellationToken ct) { - if (!IsValidName(name)) + if (!HistorianTagNames.IsValid(name)) throw new ArgumentException($"Historian tag name exceeds {MaxNameLength} characters.", nameof(name)); if (_cache.TryGetValue(name, out var cached)) return cached; await _bootstrap.EnsureAsync(ct).ConfigureAwait(false); diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.csproj b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.csproj index ff5e29c5..353f5660 100644 --- a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.csproj +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.csproj @@ -11,6 +11,8 @@ + + diff --git a/tests/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions.Tests/ZB.MOM.WW.OtOpcUa.Core.Abstractions.Tests.csproj b/tests/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions.Tests/ZB.MOM.WW.OtOpcUa.Core.Abstractions.Tests.csproj index 7c38ec29..6cae72d6 100644 --- a/tests/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions.Tests/ZB.MOM.WW.OtOpcUa.Core.Abstractions.Tests.csproj +++ b/tests/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions.Tests/ZB.MOM.WW.OtOpcUa.Core.Abstractions.Tests.csproj @@ -43,6 +43,8 @@ DriverTypeNames guard skips it — but it does hard-code OPC UA status constants, which puts it in StatusCodeParityTests' scope. --> + + diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/ProcessedReadTests.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/ProcessedReadTests.cs new file mode 100644 index 00000000..cce8846f --- /dev/null +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/ProcessedReadTests.cs @@ -0,0 +1,16 @@ +using Shouldly; +using Xunit; +using ZB.MOM.WW.OtOpcUa.Core.Abstractions; + +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests; + +/// The DB-free part of the former SQL Server integration ProcessedReadTests. +public sealed class ProcessedReadTests +{ + [Fact] + public void Every_enum_member_is_either_served_or_refused() + { + // Thirteen served; anything appended to the enum later falls to the refusal (total default). + Enum.GetValues().Count(a => AggregateFamilies.FamilyOf(a) is not null).ShouldBe(13); + } +} diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/SampleRowMapperTests.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/SampleRowMapperTests.cs new file mode 100644 index 00000000..43187cd6 --- /dev/null +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/SampleRowMapperTests.cs @@ -0,0 +1,45 @@ +using Shouldly; +using Xunit; +using ZB.MOM.WW.OtOpcUa.Core.Abstractions; + +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests; + +/// The column round-trip half of the former SQL Server SampleRowMapperTests (the TVP half stays there). +public sealed class SampleRowMapperTests +{ + public static TheoryData Values => new() + { + HistorianValue.FromDouble(12.5), + HistorianValue.FromInt64(long.MinValue + 3), + HistorianValue.FromBoolean(true), + HistorianValue.FromBoolean(false), + HistorianValue.FromString("héllo"), + HistorianValue.FromString(""), + HistorianValue.FromDateTime(new DateTime(2026, 10, 9, 1, 2, 3, DateTimeKind.Utc).AddTicks(1234567)), + HistorianValue.Empty, + }; + + [Theory] + [MemberData(nameof(Values))] + public void Every_kind_round_trips_through_the_column_quad(HistorianValue v) + { + var c = SampleColumnMapper.ToColumns(v); + SampleColumnMapper.FromColumns(c.ValueKind, c.NumValue, c.IntValue, c.StrValue).ShouldBe(v); + } + + [Fact] + public void Columns_follow_the_spec_mapping() + { + SampleColumnMapper.ToColumns(HistorianValue.FromInt64(7)).ShouldBe(new SampleColumns(2, 7d, 7L, null)); + SampleColumnMapper.ToColumns(HistorianValue.FromBoolean(true)).ShouldBe(new SampleColumns(3, 1d, 1L, null)); + SampleColumnMapper.ToColumns(HistorianValue.FromDouble(1.5)).ShouldBe(new SampleColumns(1, 1.5, null, null)); + SampleColumnMapper.ToColumns(HistorianValue.Empty).ShouldBe(new SampleColumns(0, null, null, null)); + } + + [Fact] + public void Status_is_stored_bit_for_bit() + { + SampleColumnMapper.ToDbStatus(0x80050000u).ShouldBeLessThan(0); + SampleColumnMapper.FromDbStatus(SampleColumnMapper.ToDbStatus(0x80050000u)).ShouldBe(0x80050000u); + } +} diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.Tests/StatusCodeOracleTests.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/StatusCodeOracleTests.cs similarity index 77% rename from tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.Tests/StatusCodeOracleTests.cs rename to tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/StatusCodeOracleTests.cs index 1c49e992..aa50f312 100644 --- a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.Tests/StatusCodeOracleTests.cs +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/StatusCodeOracleTests.cs @@ -1,7 +1,7 @@ using Shouldly; using Xunit; -namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.Tests; +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests; /// The pinned SDK is the oracle for the info-bit constants (StatusCodeParityTests covers the named codes). public sealed class StatusCodeOracleTests @@ -10,7 +10,7 @@ public sealed class StatusCodeOracleTests public void Calculated_bits_match_the_SDK() { var sdk = new Opc.Ua.StatusCode(Opc.Ua.StatusCodes.Good).SetAggregateBits(Opc.Ua.AggregateBits.Calculated); - sdk.Code.ShouldBe(HistorianInfoBits.WithCalculated(SqlHistorianStatusCodes.Good)); + sdk.Code.ShouldBe(HistorianInfoBits.WithCalculated(HistorianStatusCodes.Good)); ((uint)Opc.Ua.AggregateBits.Calculated).ShouldBe(HistorianInfoBits.HistorianCalculated); } @@ -18,7 +18,7 @@ public sealed class StatusCodeOracleTests public void Interpolated_bits_match_the_SDK() { var sdk = new Opc.Ua.StatusCode(Opc.Ua.StatusCodes.Good).SetAggregateBits(Opc.Ua.AggregateBits.Interpolated); - sdk.Code.ShouldBe(HistorianInfoBits.WithInterpolated(SqlHistorianStatusCodes.Good)); + sdk.Code.ShouldBe(HistorianInfoBits.WithInterpolated(HistorianStatusCodes.Good)); ((uint)Opc.Ua.AggregateBits.Interpolated).ShouldBe(HistorianInfoBits.HistorianInterpolated); } @@ -30,9 +30,9 @@ public sealed class StatusCodeOracleTests [Fact] public void Setting_one_historian_bit_clears_the_other() { - var calculatedUncertain = SqlHistorianStatusCodes.UncertainDataSubNormal | 0x400u | HistorianInfoBits.HistorianCalculated; + var calculatedUncertain = HistorianStatusCodes.UncertainDataSubNormal | 0x400u | HistorianInfoBits.HistorianCalculated; var interpolated = HistorianInfoBits.WithInterpolated(calculatedUncertain); - interpolated.ShouldBe(SqlHistorianStatusCodes.UncertainDataSubNormal | 0x400u | HistorianInfoBits.HistorianInterpolated); + interpolated.ShouldBe(HistorianStatusCodes.UncertainDataSubNormal | 0x400u | HistorianInfoBits.HistorianInterpolated); new Opc.Ua.StatusCode(interpolated).AggregateBits.ShouldBe(Opc.Ua.AggregateBits.Interpolated); var interpolatedGood = 0x400u | HistorianInfoBits.HistorianInterpolated; diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.Tests/ValueWriterPrepareTests.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/ValueWriterPrepareTests.cs similarity index 68% rename from tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.Tests/ValueWriterPrepareTests.cs rename to tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/ValueWriterPrepareTests.cs index 3f0ee8b5..77e24c06 100644 --- a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.Tests/ValueWriterPrepareTests.cs +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/ValueWriterPrepareTests.cs @@ -2,7 +2,7 @@ using Shouldly; using Xunit; using ZB.MOM.WW.OtOpcUa.Core.Abstractions; -namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.Tests; +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests; public sealed class ValueWriterPrepareTests { @@ -13,7 +13,7 @@ public sealed class ValueWriterPrepareTests { const uint uncertain = 0x40000000u; const uint badNoCommunication = 0x80310000u; - var rows = SqlServerHistorianValueWriter.Prepare( + var rows = LiveWritePreparer.Prepare( [ new(T, HistorianValue.FromDouble(double.NaN), uncertain), new(T.AddTicks(1), HistorianValue.FromDouble(double.PositiveInfinity), 0), @@ -21,26 +21,26 @@ public sealed class ValueWriterPrepareTests new(T.AddTicks(3), HistorianValue.FromDouble(1.5), 0), ], () => T); rows.Select(r => r.Value.Kind).ShouldBe([HistorianValueKind.Empty, HistorianValueKind.Empty, HistorianValueKind.Empty, HistorianValueKind.Double]); - rows.Select(r => r.StatusCode).ShouldBe([SqlHistorianStatusCodes.BadOutOfRange, SqlHistorianStatusCodes.BadOutOfRange, + rows.Select(r => r.StatusCode).ShouldBe([HistorianStatusCodes.BadOutOfRange, HistorianStatusCodes.BadOutOfRange, badNoCommunication, 0u]); } [Fact] public void A_non_bad_Empty_is_stored_BadNoData_and_a_bad_Empty_keeps_its_code() { - var rows = SqlServerHistorianValueWriter.Prepare( + var rows = LiveWritePreparer.Prepare( [ new(T, HistorianValue.Empty, 0), new(T.AddTicks(1), HistorianValue.Empty, 0x40A40000u), new(T.AddTicks(2), HistorianValue.Empty, 0x80310000u), ], () => T); - rows.Select(r => r.StatusCode).ShouldBe([SqlHistorianStatusCodes.BadNoData, SqlHistorianStatusCodes.BadNoData, 0x80310000u]); + rows.Select(r => r.StatusCode).ShouldBe([HistorianStatusCodes.BadNoData, HistorianStatusCodes.BadNoData, 0x80310000u]); } [Fact] public void Duplicate_timestamps_keep_the_first_and_null_timestamp_uses_the_clock() { - var rows = SqlServerHistorianValueWriter.Prepare( + var rows = LiveWritePreparer.Prepare( [ new(T, HistorianValue.FromInt64(1), 0), new(T, HistorianValue.FromInt64(2), 0), new(null, HistorianValue.FromInt64(3), 0), @@ -53,8 +53,8 @@ public sealed class ValueWriterPrepareTests [Fact] public void Data_type_mapping_is_one_table() { - SqlHistorianDataTypes.FromKind(HistorianValueKind.Double).ShouldBe((byte)DriverDataType.Float64); - SqlHistorianDataTypes.FromKind(HistorianValueKind.Boolean).ShouldBe((byte)DriverDataType.Boolean); - SqlHistorianDataTypes.FromKind(HistorianValueKind.Empty).ShouldBe((byte)DriverDataType.Float64); + HistorianDataTypes.FromKind(HistorianValueKind.Double).ShouldBe((byte)DriverDataType.Float64); + HistorianDataTypes.FromKind(HistorianValueKind.Boolean).ShouldBe((byte)DriverDataType.Boolean); + HistorianDataTypes.FromKind(HistorianValueKind.Empty).ShouldBe((byte)DriverDataType.Float64); } } diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests.csproj b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests.csproj new file mode 100644 index 00000000..487e0dd4 --- /dev/null +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests.csproj @@ -0,0 +1,28 @@ + + + + net10.0 + enable + enable + false + true + ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests + + + + + + + + + + all + runtime; build; native; contentfiles; analyzers; buildtransitive + + + + + + + + diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/LongNameTests.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/LongNameTests.cs index b5fa9548..1b34e04d 100644 --- a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/LongNameTests.cs +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/LongNameTests.cs @@ -1,6 +1,7 @@ using Shouldly; using Xunit; using ZB.MOM.WW.OtOpcUa.Core.Abstractions; +using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common; namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests; diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/ProcessedReadTests.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/ProcessedReadTests.cs index 841ffb1a..bc98a74c 100644 --- a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/ProcessedReadTests.cs +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/ProcessedReadTests.cs @@ -360,13 +360,6 @@ public sealed class ProcessedReadTests(SqlHistorianFixture fx) src.GetHealthSnapshot().TotalQueries.ShouldBe(0); } - [Fact] - public void Every_enum_member_is_either_served_or_refused() - { - // Thirteen served; anything appended to the enum later falls to the refusal (total default). - Enum.GetValues().Count(a => SqlAggregateQueries.FamilyOf(a) is not null).ShouldBe(13); - } - [Fact] public async Task The_bucket_count_is_capped_at_100000() { diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/QualityContractTests.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/QualityContractTests.cs index 7666d8c2..c08a856a 100644 --- a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/QualityContractTests.cs +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/QualityContractTests.cs @@ -1,6 +1,7 @@ using Shouldly; using Xunit; using ZB.MOM.WW.OtOpcUa.Core.Abstractions; +using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common; namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests; @@ -96,8 +97,8 @@ public sealed class QualityContractTests(SqlHistorianFixture fx) new(At(2), HistorianValue.Empty, BadNoCommunication), new(At(3), HistorianValue.FromDouble(1), 0u), ], Ct); - r.ShouldBe([BadInvalidArgument, BadInvalidArgument, SqlHistorianStatusCodes.GoodEntryInserted, - SqlHistorianStatusCodes.GoodEntryInserted]); + r.ShouldBe([BadInvalidArgument, BadInvalidArgument, HistorianStatusCodes.GoodEntryInserted, + HistorianStatusCodes.GoodEntryInserted]); (await db.Source().ReadRawAsync("u", At(0), At(10), 0, Ct)).Samples .Select(s => s.StatusCode).ShouldBe([BadNoCommunication, 0u]); } diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/RawReadTests.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/RawReadTests.cs index 531d3473..8d62d081 100644 --- a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/RawReadTests.cs +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/RawReadTests.cs @@ -2,6 +2,7 @@ using Microsoft.Data.SqlClient; using Shouldly; using Xunit; using ZB.MOM.WW.OtOpcUa.Core.Abstractions; +using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common; namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests; @@ -59,7 +60,7 @@ internal sealed class ReadTestDb } foreach (var (t, v, status) in samples) { - var col = SampleRowMapper.ToColumns(v); + var col = SampleColumnMapper.ToColumns(v); await using var cmd = Conn.CreateCommand(c, $"INSERT {Conn.Q("Sample")} (TagId, SourceTimeUtc, ValueKind, NumValue, IntValue, StrValue, StatusCode, ServerTimeUtc) " + "VALUES (@id, @t, @k, @num, @int, @str, @st, @srv)"); @@ -69,7 +70,7 @@ internal sealed class ReadTestDb cmd.Parameters.Add(new SqlParameter("@num", System.Data.SqlDbType.Float) { Value = (object?)col.NumValue ?? DBNull.Value }); cmd.Parameters.Add(new SqlParameter("@int", System.Data.SqlDbType.BigInt) { Value = (object?)col.IntValue ?? DBNull.Value }); cmd.Parameters.Add(new SqlParameter("@str", System.Data.SqlDbType.NVarChar, -1) { Value = (object?)col.StrValue ?? DBNull.Value }); - cmd.Parameters.AddWithValue("@st", SampleRowMapper.ToDbStatus(status)); + cmd.Parameters.AddWithValue("@st", SampleColumnMapper.ToDbStatus(status)); SqlHistorianConnection.AddDateTime2(cmd, "@srv", t.AddSeconds(1)); await cmd.ExecuteNonQueryAsync(); } @@ -253,7 +254,7 @@ public sealed class RawReadTests(SqlHistorianFixture fx) // Another node creates the schema; the same data source then succeeds (bootstrap failure is not latched). var creator = new SqlServerHistorianOptions { ConnectionString = o.ConnectionString, Schema = o.Schema }; await new SqlServerSchemaBootstrapper(new SqlHistorianConnection(creator), creator).EnsureAsync(Ct); - clock.Now += SqlServerSchemaBootstrapper.FailureCacheDuration; // past the 5 s failure cache (M-2) + clock.Now += BootstrapFailureCache.DefaultDuration; // past the 5 s failure cache (M-2) (await src.ReadRawAsync("A", At(0), At(1), 0, Ct)).Samples.ShouldBeEmpty(); src.GetHealthSnapshot().ConsecutiveFailures.ShouldBe(0); } diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/SchemaBootstrapIntegrationTests.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/SchemaBootstrapIntegrationTests.cs index 9f90db17..6fd3ca86 100644 --- a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/SchemaBootstrapIntegrationTests.cs +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/SchemaBootstrapIntegrationTests.cs @@ -1,6 +1,7 @@ using Microsoft.Data.SqlClient; using Shouldly; using Xunit; +using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common; namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests; @@ -82,12 +83,12 @@ public sealed class SchemaBootstrapIntegrationTests(SqlHistorianFixture fx) var checker = new SqlServerSchemaBootstrapper(new SqlHistorianConnection(o), o, timeProvider: clock); await Should.ThrowAsync(() => checker.EnsureAsync(CancellationToken.None)); await Boot(Opts(s)).EnsureAsync(CancellationToken.None); - clock.Now += SqlServerSchemaBootstrapper.FailureCacheDuration; + clock.Now += BootstrapFailureCache.DefaultDuration; await checker.EnsureAsync(CancellationToken.None); // same instance retries once the failure cache expires } /// - /// M-2: within of a failure the bootstrapper + /// M-2: within of a failure the bootstrapper /// re-throws it without touching SQL — proven by creating the schema in between: a call that went to SQL /// would now pass. /// @@ -103,7 +104,7 @@ public sealed class SchemaBootstrapIntegrationTests(SqlHistorianFixture fx) await Boot(Opts(s)).EnsureAsync(CancellationToken.None); // the schema now exists - clock.Now += SqlServerSchemaBootstrapper.FailureCacheDuration - TimeSpan.FromTicks(1); + clock.Now += BootstrapFailureCache.DefaultDuration - TimeSpan.FromTicks(1); await Should.ThrowAsync(() => checker.EnsureAsync(CancellationToken.None)); clock.Now += TimeSpan.FromTicks(1); @@ -156,7 +157,7 @@ public sealed class SchemaBootstrapIntegrationTests(SqlHistorianFixture fx) "SELECT @id, SourceTimeUtc, ValueKind, NumValue, IntValue, StrValue, StatusCode, SYSUTCDATETIME() FROM @rows"); cmd.Parameters.AddWithValue("@id", tagId); conn.AddStructured(cmd, "@rows", "SampleRow", - SampleRowMapper.ToSampleRecords([new SqlSampleRow(t, ZB.MOM.WW.OtOpcUa.Core.Abstractions.HistorianValue.FromDouble(1), 0u)])); + SampleRowMapper.ToSampleRecords([new HistorianSampleRow(t, ZB.MOM.WW.OtOpcUa.Core.Abstractions.HistorianValue.FromDouble(1), 0u)])); await cmd.ExecuteNonQueryAsync(); await using var q = conn.CreateCommand(c, $"SELECT SourceTimeUtc FROM {conn.Q("Sample")}"); await using var r = await q.ExecuteReaderAsync(); diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/UpdateTests.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/UpdateTests.cs index 670aeead..98001164 100644 --- a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/UpdateTests.cs +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/UpdateTests.cs @@ -2,14 +2,15 @@ using Microsoft.Data.SqlClient; using Shouldly; using Xunit; using ZB.MOM.WW.OtOpcUa.Core.Abstractions; +using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common; namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests; [Collection(SqlHistorianCollection.Name)] public sealed class UpdateTests(SqlHistorianFixture fx) { - private const uint Good = SqlHistorianStatusCodes.Good; - private const uint UncertainStatus = SqlHistorianStatusCodes.UncertainDataSubNormal; + private const uint Good = HistorianStatusCodes.Good; + private const uint UncertainStatus = HistorianStatusCodes.UncertainDataSubNormal; // Deliberately NOT millisecond aligned: sub-ms ticks must survive the TVP, MERGE join and DELETE join. private static readonly DateTime T0 = new(2026, 1, 1, 0, 0, 0, DateTimeKind.Utc); @@ -58,10 +59,10 @@ public sealed class UpdateTests(SqlHistorianFixture fx) { var (u, s) = Make(); var r1 = await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T(1), 1), V(T(2), 2)], Ct); - r1.ShouldBe([SqlHistorianStatusCodes.GoodEntryInserted, SqlHistorianStatusCodes.GoodEntryInserted]); + r1.ShouldBe([HistorianStatusCodes.GoodEntryInserted, HistorianStatusCodes.GoodEntryInserted]); // Mixed: T(2) exists, T(3) new, T(1) exists - out of timestamp order on purpose. var r2 = await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T(3), 3), V(T(2), 99), V(T(1), 98)], Ct); - r2.ShouldBe([SqlHistorianStatusCodes.GoodEntryInserted, SqlHistorianStatusCodes.BadEntryExists, SqlHistorianStatusCodes.BadEntryExists]); + r2.ShouldBe([HistorianStatusCodes.GoodEntryInserted, HistorianStatusCodes.BadEntryExists, HistorianStatusCodes.BadEntryExists]); var rows = await RowsAsync(s, "t"); rows.Select(x => x.Num).ShouldBe([1d, 2d, 3d]); // existing values untouched rows.Select(x => x.Time).ShouldBe([T(1), T(2), T(3)]); // sub-ms ticks preserved @@ -74,7 +75,7 @@ public sealed class UpdateTests(SqlHistorianFixture fx) await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T(1), 1), V(T(2), 2)], Ct); var r = await u.UpdateDataAsync("t", HistoryUpdateMode.Replace, [V(T(2), 22, UncertainStatus), V(T(5), 5), V(T(1), 11)], Ct); - r.ShouldBe([SqlHistorianStatusCodes.GoodEntryReplaced, SqlHistorianStatusCodes.BadNoEntryExists, SqlHistorianStatusCodes.GoodEntryReplaced]); + r.ShouldBe([HistorianStatusCodes.GoodEntryReplaced, HistorianStatusCodes.BadNoEntryExists, HistorianStatusCodes.GoodEntryReplaced]); var rows = await RowsAsync(s, "t"); rows.Count.ShouldBe(2); // T(5) was not created rows[0].Num.ShouldBe(11d); @@ -89,7 +90,7 @@ public sealed class UpdateTests(SqlHistorianFixture fx) var (u, s) = Make(); await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T(1), 1)], Ct); var r = await u.UpdateDataAsync("t", HistoryUpdateMode.Update, [V(T(2), 2), V(T(1), 10)], Ct); - r.ShouldBe([SqlHistorianStatusCodes.GoodEntryInserted, SqlHistorianStatusCodes.GoodEntryReplaced]); + r.ShouldBe([HistorianStatusCodes.GoodEntryInserted, HistorianStatusCodes.GoodEntryReplaced]); (await RowsAsync(s, "t")).Select(x => x.Num).ShouldBe([10d, 2d]); } @@ -98,13 +99,13 @@ public sealed class UpdateTests(SqlHistorianFixture fx) { var (u, s) = Make(); (await u.UpdateDataAsync("rep", HistoryUpdateMode.Replace, [V(T(1), 1)], Ct)) - .ShouldBe([SqlHistorianStatusCodes.BadNoEntryExists]); - (await u.DeleteRawAsync("del", T(0), T(9), Ct)).ShouldBe(SqlHistorianStatusCodes.BadNoData); - (await u.DeleteAtTimeAsync("del2", [T(1)], Ct)).ShouldBe([SqlHistorianStatusCodes.BadNoEntryExists]); + .ShouldBe([HistorianStatusCodes.BadNoEntryExists]); + (await u.DeleteRawAsync("del", T(0), T(9), Ct)).ShouldBe(HistorianStatusCodes.BadNoData); + (await u.DeleteAtTimeAsync("del2", [T(1)], Ct)).ShouldBe([HistorianStatusCodes.BadNoEntryExists]); (await TagCountAsync(s)).ShouldBe(0); // none of those created a tag - (await u.UpdateDataAsync("ins", HistoryUpdateMode.Insert, [V(T(1), 1)], Ct)).ShouldBe([SqlHistorianStatusCodes.GoodEntryInserted]); - (await u.UpdateDataAsync("upd", HistoryUpdateMode.Update, [V(T(1), 1)], Ct)).ShouldBe([SqlHistorianStatusCodes.GoodEntryInserted]); + (await u.UpdateDataAsync("ins", HistoryUpdateMode.Insert, [V(T(1), 1)], Ct)).ShouldBe([HistorianStatusCodes.GoodEntryInserted]); + (await u.UpdateDataAsync("upd", HistoryUpdateMode.Update, [V(T(1), 1)], Ct)).ShouldBe([HistorianStatusCodes.GoodEntryInserted]); (await TagCountAsync(s)).ShouldBe(2); } @@ -114,13 +115,13 @@ public sealed class UpdateTests(SqlHistorianFixture fx) var (u, s) = Make(); var r = await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(null, 1), V(T(1), 2), V(DateTime.MinValue, 3)], Ct); - r.ShouldBe([SqlHistorianStatusCodes.BadInvalidTimestamp, SqlHistorianStatusCodes.GoodEntryInserted, SqlHistorianStatusCodes.BadInvalidTimestamp]); + r.ShouldBe([HistorianStatusCodes.BadInvalidTimestamp, HistorianStatusCodes.GoodEntryInserted, HistorianStatusCodes.BadInvalidTimestamp]); (await RowsAsync(s, "t")).Count.ShouldBe(1); // All-invalid request: no DB work, no tag created, still one code per item. var (u2, s2) = Make(); await u2.UpdateDataAsync("seed", HistoryUpdateMode.Insert, [V(T(1), 1)], Ct); // bootstraps the schema - (await u2.UpdateDataAsync("x", HistoryUpdateMode.Insert, [V(null, 1)], Ct)).ShouldBe([SqlHistorianStatusCodes.BadInvalidTimestamp]); + (await u2.UpdateDataAsync("x", HistoryUpdateMode.Insert, [V(null, 1)], Ct)).ShouldBe([HistorianStatusCodes.BadInvalidTimestamp]); (await TagCountAsync(s2)).ShouldBe(1); // only "seed"; "x" was never created } @@ -130,8 +131,8 @@ public sealed class UpdateTests(SqlHistorianFixture fx) var (u, s) = Make(); var r = await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T(1), double.NaN), V(T(2), 2), V(T(3), double.PositiveInfinity), V(T(4), double.NegativeInfinity)], Ct); - r.ShouldBe([SqlHistorianUpdateStatusCodes.BadInvalidArgument, SqlHistorianStatusCodes.GoodEntryInserted, - SqlHistorianUpdateStatusCodes.BadInvalidArgument, SqlHistorianUpdateStatusCodes.BadInvalidArgument]); + r.ShouldBe([HistorianStatusCodes.BadInvalidArgument, HistorianStatusCodes.GoodEntryInserted, + HistorianStatusCodes.BadInvalidArgument, HistorianStatusCodes.BadInvalidArgument]); (await RowsAsync(s, "t")).Select(x => x.Num).ShouldBe([2d]); } @@ -140,7 +141,7 @@ public sealed class UpdateTests(SqlHistorianFixture fx) { var (u, s) = Make(); var r = await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T(1), 1), V(T(2), 2), V(T(1), 99)], Ct); - r.ShouldBe([SqlHistorianStatusCodes.GoodEntryInserted, SqlHistorianStatusCodes.GoodEntryInserted, SqlHistorianStatusCodes.BadEntryExists]); + r.ShouldBe([HistorianStatusCodes.GoodEntryInserted, HistorianStatusCodes.GoodEntryInserted, HistorianStatusCodes.BadEntryExists]); (await RowsAsync(s, "t")).Select(x => x.Num).ShouldBe([1d, 2d]); } @@ -150,11 +151,11 @@ public sealed class UpdateTests(SqlHistorianFixture fx) var (u, s) = Make(); await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T(1), 1)], Ct); var rep = await u.UpdateDataAsync("t", HistoryUpdateMode.Replace, [V(T(1), 5), V(T(1), 6)], Ct); - rep.ShouldBe([SqlHistorianStatusCodes.GoodEntryReplaced, SqlHistorianStatusCodes.GoodEntryReplaced]); + rep.ShouldBe([HistorianStatusCodes.GoodEntryReplaced, HistorianStatusCodes.GoodEntryReplaced]); (await RowsAsync(s, "t")).Single().Num.ShouldBe(6d); var upd = await u.UpdateDataAsync("t", HistoryUpdateMode.Update, [V(T(7), 7), V(T(7), 8)], Ct); - upd.ShouldBe([SqlHistorianStatusCodes.GoodEntryInserted, SqlHistorianStatusCodes.GoodEntryInserted]); + upd.ShouldBe([HistorianStatusCodes.GoodEntryInserted, HistorianStatusCodes.GoodEntryInserted]); (await RowsAsync(s, "t")).Last().Num.ShouldBe(8d); } @@ -164,7 +165,7 @@ public sealed class UpdateTests(SqlHistorianFixture fx) var (u, s) = Make(); await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [new HistorizationValue(T(1), HistorianValue.FromDouble(1.5), Good)], Ct); (await u.UpdateDataAsync("t", HistoryUpdateMode.Replace, - [new HistorizationValue(T(1), HistorianValue.FromString("hi"), Good)], Ct)).ShouldBe([SqlHistorianStatusCodes.GoodEntryReplaced]); + [new HistorizationValue(T(1), HistorianValue.FromString("hi"), Good)], Ct)).ShouldBe([HistorianStatusCodes.GoodEntryReplaced]); var rows = await RowsAsync(s, "t"); rows.Single().Num.ShouldBeNull(); // NumValue cleared by the String replace } @@ -177,9 +178,9 @@ public sealed class UpdateTests(SqlHistorianFixture fx) // [T(1), T(3)) removes T(1) and T(2); T(3) (the end) survives. (await u.DeleteRawAsync("t", T(1), T(3), Ct)).ShouldBe(Good); (await RowsAsync(s, "t")).Select(x => x.Time).ShouldBe([T(3)]); - (await u.DeleteRawAsync("t", T(1), T(3), Ct)).ShouldBe(SqlHistorianStatusCodes.BadNoData); + (await u.DeleteRawAsync("t", T(1), T(3), Ct)).ShouldBe(HistorianStatusCodes.BadNoData); // sub-ms boundary: a start one tick after T(3) excludes it. - (await u.DeleteRawAsync("t", T(3).AddTicks(1), T(3).AddTicks(100), Ct)).ShouldBe(SqlHistorianStatusCodes.BadNoData); + (await u.DeleteRawAsync("t", T(3).AddTicks(1), T(3).AddTicks(100), Ct)).ShouldBe(HistorianStatusCodes.BadNoData); (await RowsAsync(s, "t")).Count.ShouldBe(1); } @@ -189,7 +190,7 @@ public sealed class UpdateTests(SqlHistorianFixture fx) var (u, s) = Make(); await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T(1), 1), V(T(2), 2), V(T(3), 3)], Ct); var r = await u.DeleteAtTimeAsync("t", [T(3), T(2).AddTicks(1) /* near miss */, T(1), DateTime.MinValue], Ct); - r.ShouldBe([Good, SqlHistorianStatusCodes.BadNoEntryExists, Good, SqlHistorianStatusCodes.BadInvalidTimestamp]); + r.ShouldBe([Good, HistorianStatusCodes.BadNoEntryExists, Good, HistorianStatusCodes.BadInvalidTimestamp]); (await RowsAsync(s, "t")).Select(x => x.Time).ShouldBe([T(2)]); } @@ -198,7 +199,7 @@ public sealed class UpdateTests(SqlHistorianFixture fx) { var (u, s) = Make(); await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T(1), 1)], Ct); - (await u.DeleteAtTimeAsync("t", [T(1), T(1)], Ct)).ShouldBe([Good, SqlHistorianStatusCodes.BadNoEntryExists]); + (await u.DeleteAtTimeAsync("t", [T(1), T(1)], Ct)).ShouldBe([Good, HistorianStatusCodes.BadNoEntryExists]); (await RowsAsync(s, "t")).ShouldBeEmpty(); } } diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/WriteTests.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/WriteTests.cs index 9b8a4ac1..3945257e 100644 --- a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/WriteTests.cs +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/WriteTests.cs @@ -3,6 +3,7 @@ using Microsoft.Data.SqlClient; using Shouldly; using Xunit; using ZB.MOM.WW.OtOpcUa.Core.Abstractions; +using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common; namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests; @@ -55,11 +56,11 @@ public sealed class WriteTests(SqlHistorianFixture fx) var rows = await RowsAsync(s, "tag.a"); rows.Count.ShouldBe(6); for (var i = 0; i < 6; i++) rows[i].T.Ticks.ShouldBe(vals[i].TimestampUtc!.Value.Ticks); - var back = rows.Select(r => SampleRowMapper.FromColumns(r.Kind, r.N, r.I, r.S)).ToList(); + var back = rows.Select(r => SampleColumnMapper.FromColumns(r.Kind, r.N, r.I, r.S)).ToList(); back.ShouldBe(vals.Select(v => v.Value).ToList()); (rows[5].N, rows[5].I, rows[5].S).ShouldBe((null, null, null)); rows[1].Status.ShouldBe(unchecked((int)0x40000000u)); - SampleRowMapper.FromDbStatus(rows[5].Status).ShouldBe(0x80310000u); + SampleColumnMapper.FromDbStatus(rows[5].Status).ShouldBe(0x80310000u); } [Fact] @@ -101,8 +102,8 @@ public sealed class WriteTests(SqlHistorianFixture fx) rows.Count.ShouldBe(3); rows[0].Kind.ShouldBe((byte)HistorianValueKind.Empty); rows[0].N.ShouldBeNull(); - rows[0].Status.ShouldBe(unchecked((int)SqlHistorianStatusCodes.BadOutOfRange)); // Uncertain NaN ⇒ Bad (I-1) - rows[1].Status.ShouldBe(unchecked((int)SqlHistorianStatusCodes.BadOutOfRange)); // Good +Inf ⇒ Bad (I-1) + rows[0].Status.ShouldBe(unchecked((int)HistorianStatusCodes.BadOutOfRange)); // Uncertain NaN ⇒ Bad (I-1) + rows[1].Status.ShouldBe(unchecked((int)HistorianStatusCodes.BadOutOfRange)); // Good +Inf ⇒ Bad (I-1) rows[2].N.ShouldBe(7); } diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.Tests/AlarmEventIdTests.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.Tests/AlarmEventIdTests.cs index 03b12e75..3050398e 100644 --- a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.Tests/AlarmEventIdTests.cs +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.Tests/AlarmEventIdTests.cs @@ -2,6 +2,7 @@ using Shouldly; using Xunit; using ZB.MOM.WW.OtOpcUa.Core.Abstractions; using ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian; +using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common; namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.Tests; @@ -16,8 +17,8 @@ public sealed class AlarmEventIdTests [Fact] public void Id_is_stable_lowercase_hex_sha256() { - var a = SqlServerAlarmHistorianWriter.ComputeEventId(Ev()); - a.ShouldBe(SqlServerAlarmHistorianWriter.ComputeEventId(Ev())); + var a = AlarmEventIds.ComputeEventId(Ev()); + a.ShouldBe(AlarmEventIds.ComputeEventId(Ev())); a.Length.ShouldBe(64); a.ShouldBe(a.ToLowerInvariant()); } @@ -26,23 +27,23 @@ public sealed class AlarmEventIdTests public void Field_boundaries_are_unambiguous() { // The delimiter-based scheme collided on these; length prefixes must not. - SqlServerAlarmHistorianWriter.ComputeEventId(Ev(path: "a|b", id: "c")) - .ShouldNotBe(SqlServerAlarmHistorianWriter.ComputeEventId(Ev(path: "a", id: "b|c"))); - SqlServerAlarmHistorianWriter.ComputeEventId(Ev(path: "ab", id: "c")) - .ShouldNotBe(SqlServerAlarmHistorianWriter.ComputeEventId(Ev(path: "a", id: "bc"))); + AlarmEventIds.ComputeEventId(Ev(path: "a|b", id: "c")) + .ShouldNotBe(AlarmEventIds.ComputeEventId(Ev(path: "a", id: "b|c"))); + AlarmEventIds.ComputeEventId(Ev(path: "ab", id: "c")) + .ShouldNotBe(AlarmEventIds.ComputeEventId(Ev(path: "a", id: "bc"))); } [Fact] public void Null_and_empty_differ_and_every_field_participates() { - var basis = SqlServerAlarmHistorianWriter.ComputeEventId(Ev()); - SqlServerAlarmHistorianWriter.ComputeEventId(Ev(comment: null)).ShouldNotBe(SqlServerAlarmHistorianWriter.ComputeEventId(Ev(comment: ""))); + var basis = AlarmEventIds.ComputeEventId(Ev()); + AlarmEventIds.ComputeEventId(Ev(comment: null)).ShouldNotBe(AlarmEventIds.ComputeEventId(Ev(comment: ""))); foreach (var other in new[] { Ev(path: "Q"), Ev(id: "B"), Ev(kind: "Cleared"), Ev(user: "v"), Ev(comment: "d"), Ev(msg: "x"), Ev(t: T.AddTicks(1)), }) - SqlServerAlarmHistorianWriter.ComputeEventId(other).ShouldNotBe(basis); + AlarmEventIds.ComputeEventId(other).ShouldNotBe(basis); } [Theory] diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.Tests/SampleRowMapperTests.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.Tests/SampleRowMapperTests.cs index c71aefe5..909b3810 100644 --- a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.Tests/SampleRowMapperTests.cs +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.Tests/SampleRowMapperTests.cs @@ -2,52 +2,18 @@ using Microsoft.Data.SqlClient.Server; using Shouldly; using Xunit; using ZB.MOM.WW.OtOpcUa.Core.Abstractions; +using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common; namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.Tests; +/// The TVP half of the mapper tests (the column round-trip half moved to Sql.Common.Tests). public sealed class SampleRowMapperTests { - public static TheoryData Values => new() - { - HistorianValue.FromDouble(12.5), - HistorianValue.FromInt64(long.MinValue + 3), - HistorianValue.FromBoolean(true), - HistorianValue.FromBoolean(false), - HistorianValue.FromString("héllo"), - HistorianValue.FromString(""), - HistorianValue.FromDateTime(new DateTime(2026, 10, 9, 1, 2, 3, DateTimeKind.Utc).AddTicks(1234567)), - HistorianValue.Empty, - }; - - [Theory] - [MemberData(nameof(Values))] - public void Every_kind_round_trips_through_the_column_quad(HistorianValue v) - { - var c = SampleRowMapper.ToColumns(v); - SampleRowMapper.FromColumns(c.ValueKind, c.NumValue, c.IntValue, c.StrValue).ShouldBe(v); - } - - [Fact] - public void Columns_follow_the_spec_mapping() - { - SampleRowMapper.ToColumns(HistorianValue.FromInt64(7)).ShouldBe(new SampleColumns(2, 7d, 7L, null)); - SampleRowMapper.ToColumns(HistorianValue.FromBoolean(true)).ShouldBe(new SampleColumns(3, 1d, 1L, null)); - SampleRowMapper.ToColumns(HistorianValue.FromDouble(1.5)).ShouldBe(new SampleColumns(1, 1.5, null, null)); - SampleRowMapper.ToColumns(HistorianValue.Empty).ShouldBe(new SampleColumns(0, null, null, null)); - } - - [Fact] - public void Status_is_stored_bit_for_bit() - { - SampleRowMapper.ToDbStatus(0x80050000u).ShouldBeLessThan(0); - SampleRowMapper.FromDbStatus(SampleRowMapper.ToDbStatus(0x80050000u)).ShouldBe(0x80050000u); - } - [Fact] public void Sample_records_keep_sub_millisecond_ticks() { var t = new DateTime(2026, 10, 9, 0, 0, 0, DateTimeKind.Utc).AddTicks(1234567); - var rec = SampleRowMapper.ToSampleRecords([new SqlSampleRow(t, HistorianValue.FromDouble(2), 0x80050000u)]).Single(); + var rec = SampleRowMapper.ToSampleRecords([new HistorianSampleRow(t, HistorianValue.FromDouble(2), 0x80050000u)]).Single(); rec.GetDateTime(0).Ticks.ShouldBe(t.Ticks); rec.GetByte(1).ShouldBe((byte)1); rec.GetDouble(2).ShouldBe(2d); From f0700571e8fa239139cae797cb3f0f13434bada6 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Sun, 11 Oct 2026 01:35:51 -0400 Subject: [PATCH 07/23] test(historian): direct Sql.Common evaluator tests (1b) 51 new test cases (34 methods) in Sql.Common.Tests covering AtTimeEvaluator, AggregateEvaluator (hand-built tag-A bucket rows, ProcessedRead literals), BucketPlan, LiveWritePreparer and HistoryUpdatePlanner (incl. resolutionTicks=10 truncate-before-dedup and the superseded-duplicate outcome), plus EventContinuation, HistorianHealthCounters, ClampSeverity and StoredSample.FromStored. Each was falsified by reverting the rule it pins (37 mutations, all red, all restored). Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe --- .../AggregateEvaluatorTests.cs | 122 ++++++++++++++++++ .../AtTimeEvaluatorTests.cs | 109 ++++++++++++++++ .../BucketPlanTests.cs | 71 ++++++++++ .../Fixture.cs | 37 ++++++ .../HistoryUpdatePlannerTests.cs | 111 ++++++++++++++++ .../LiveWritePreparerTests.cs | 37 ++++++ .../SharedHelperTests.cs | 67 ++++++++++ 7 files changed, 554 insertions(+) create mode 100644 tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/AggregateEvaluatorTests.cs create mode 100644 tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/AtTimeEvaluatorTests.cs create mode 100644 tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/BucketPlanTests.cs create mode 100644 tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/Fixture.cs create mode 100644 tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/HistoryUpdatePlannerTests.cs create mode 100644 tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/LiveWritePreparerTests.cs create mode 100644 tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/SharedHelperTests.cs diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/AggregateEvaluatorTests.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/AggregateEvaluatorTests.cs new file mode 100644 index 00000000..4e9270e5 --- /dev/null +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/AggregateEvaluatorTests.cs @@ -0,0 +1,122 @@ +using Shouldly; +using Xunit; +using ZB.MOM.WW.OtOpcUa.Core.Abstractions; +using static ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests.Fixture; + +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests; + +/// +/// Direct tests of the aggregate evaluator over hand-built bucket rows for tag A, 20 s intervals from T0, window +/// [t0, t0+80s): B0 = {10, 20}; B1 = {Bad 30, 40}; B2 = {50}; B3 empty. Literals from the SqlServer +/// ProcessedReadTests. +/// +public sealed class AggregateEvaluatorTests +{ + private static readonly DateTime Now = At(3600); + + private static Dictionary RowsA(bool maximum = false) => new() + { + [0] = new BucketRow + { + Bucket = 0, Total = 2, Usable = 2, Good = 2, Bad = 0, NonNumeric = 0, Avg = 15, Min = 10, Max = 20, Sum = 30, + First = HistorianValue.FromDouble(10), Last = HistorianValue.FromDouble(20), + Extreme = maximum ? 20 : 10, ExtremeTime = maximum ? At(10) : At(0), + }, + [1] = new BucketRow + { + Bucket = 1, Total = 2, Usable = 1, Good = 1, Bad = 1, NonNumeric = 0, Avg = 40, Min = 40, Max = 40, Sum = 40, + First = HistorianValue.FromDouble(40), Last = HistorianValue.FromDouble(40), Extreme = 40, ExtremeTime = At(30), + }, + [2] = new BucketRow + { + Bucket = 2, Total = 1, Usable = 1, Good = 1, Bad = 0, NonNumeric = 0, Avg = 50, Min = 50, Max = 50, Sum = 50, + First = HistorianValue.FromDouble(50), Last = HistorianValue.FromDouble(50), Extreme = 50, ExtremeTime = At(40), + }, + }; + + private static IReadOnlyList ReadA(HistoryAggregateType agg, bool maximum = false) + { + var plan = BucketPlan.Create(At(0), At(80), TimeSpan.FromSeconds(20)); + var evaluated = RowsA(maximum).ToDictionary(kv => kv.Key, + kv => AggregateEvaluator.Evaluate(agg, kv.Value, plan.BucketStart(kv.Key), Now)); + return plan.Assemble(evaluated, t => AggregateEvaluator.NoData(t, Now)); + } + + [Theory] + [InlineData(HistoryAggregateType.Average, 15d, 40d, 50d)] + [InlineData(HistoryAggregateType.Minimum, 10d, 40d, 50d)] + [InlineData(HistoryAggregateType.Maximum, 20d, 40d, 50d)] + [InlineData(HistoryAggregateType.Total, 30d, 40d, 50d)] + [InlineData(HistoryAggregateType.Start, 10d, 40d, 50d)] + [InlineData(HistoryAggregateType.End, 20d, 40d, 50d)] + [InlineData(HistoryAggregateType.Delta, 10d, 0d, 0d)] + [InlineData(HistoryAggregateType.Range, 10d, 0d, 0d)] + public void Numeric_aggregates_over_hand_built_bucket_rows(HistoryAggregateType agg, double b0, double b1, double b2) + { + var s = ReadA(agg); + s.Count.ShouldBe(4); + s.Select(x => x.Value).ShouldBe([b0, b1, b2, null]); + s.Select(x => x.StatusCode).ShouldBe([GoodCalc, UncertainCalc, GoodCalc, NoDataCalc]); + s.Select(x => x.SourceTimestampUtc).ShouldBe([At(0), At(20), At(40), At(60)]); + } + + [Fact] + public void Count_counts_usable_samples_as_int() + { + var s = ReadA(HistoryAggregateType.Count); + s.Select(x => x.Value).ShouldBe([(object?)2, 1, 1, null]); + s.Select(x => x.StatusCode).ShouldBe([GoodCalc, UncertainCalc, GoodCalc, NoDataCalc]); + } + + [Theory] + [InlineData(HistoryAggregateType.PercentGood, 100d, 50d, 100d)] + [InlineData(HistoryAggregateType.PercentBad, 0d, 50d, 0d)] + public void Percent_aggregates_count_every_sample_and_stay_good(HistoryAggregateType agg, double b0, double b1, double b2) + { + var s = ReadA(agg); + s.Select(x => x.Value).ShouldBe([b0, b1, b2, null]); + s.Select(x => x.StatusCode).ShouldBe([GoodCalc, GoodCalc, GoodCalc, NoDataCalc]); + } + + [Fact] + public void Actual_time_aggregates_stamp_the_sample_time() + { + var min = ReadA(HistoryAggregateType.MinimumActualTime); + min.Select(x => x.Value).ShouldBe([10d, 40d, 50d, null]); + min.Select(x => x.SourceTimestampUtc).ShouldBe([At(0), At(30), At(40), At(60)]); + min.Select(x => x.StatusCode).ShouldBe([GoodCalc, UncertainCalc, GoodCalc, NoDataCalc]); + var max = ReadA(HistoryAggregateType.MaximumActualTime, maximum: true); + max.Select(x => x.Value).ShouldBe([20d, 40d, 50d, null]); + max.Select(x => x.SourceTimestampUtc).ShouldBe([At(10), At(30), At(40), At(60)]); + } + + [Fact] + public void A_usable_non_numeric_sample_makes_a_numeric_aggregate_invalid_inputs_but_not_count() + { + var row = new BucketRow { Total = 2, Usable = 2, Good = 2, NonNumeric = 1, Avg = 1, First = HistorianValue.FromString("x"), + Last = HistorianValue.FromDouble(1) }; + var avg = AggregateEvaluator.Evaluate(HistoryAggregateType.Average, row, At(0), Now); + avg.Value.ShouldBeNull(); + avg.StatusCode.ShouldBe(InvalidInputsCalc); + AggregateEvaluator.Evaluate(HistoryAggregateType.Count, row, At(0), Now).Value.ShouldBe(2); + AggregateEvaluator.Evaluate(HistoryAggregateType.Delta, row, At(0), Now).StatusCode.ShouldBe(InvalidInputsCalc); + } + + [Fact] + public void A_bucket_with_only_bad_samples_is_no_data_except_for_the_percentages() + { + var row = new BucketRow { Total = 2, Usable = 0, Good = 0, Bad = 2 }; + var avg = AggregateEvaluator.Evaluate(HistoryAggregateType.Average, row, At(0), Now); + avg.Value.ShouldBeNull(); + avg.StatusCode.ShouldBe(NoDataCalc); + AggregateEvaluator.Evaluate(HistoryAggregateType.PercentBad, row, At(0), Now).Value.ShouldBe(100d); + } + + [Fact] + public void An_unsupported_aggregate_throws_not_supported() + { + var ex = Should.Throw(() => + AggregateEvaluator.Evaluate(HistoryAggregateType.TimeAverage, RowsA()[0], At(0), Now)); + ex.Aggregate.ShouldBe(HistoryAggregateType.TimeAverage); + } +} diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/AtTimeEvaluatorTests.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/AtTimeEvaluatorTests.cs new file mode 100644 index 00000000..5ab37704 --- /dev/null +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/AtTimeEvaluatorTests.cs @@ -0,0 +1,109 @@ +using Shouldly; +using Xunit; +using ZB.MOM.WW.OtOpcUa.Core.Abstractions; +using static ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests.Fixture; + +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests; + +/// Direct tests of the AtTime rules, with the literals of the SqlServer AtTimeReadTests. +public sealed class AtTimeEvaluatorTests +{ + private static readonly DateTime Now = At(3600); + + [Fact] + public void AtTime_inside_an_outage_returns_the_bad_neighbours_code_with_no_value() + { + var t0 = new DateTime(2026, 10, 9, 8, 0, 0, DateTimeKind.Utc); + var prev = new StoredSample(t0, HistorianValue.Empty, 0x80310000u); + var next = new StoredSample(t0.AddSeconds(10), HistorianValue.FromDouble(5), 0); + var r = AtTimeEvaluator.Evaluate(t0.AddSeconds(5), prev, next, t0, t0.AddHours(1)); + r.StatusCode.ShouldBe(0x80310000u); + r.Value.ShouldBeNull(); + } + + [Fact] + public void A_bad_valued_previous_neighbour_keeps_its_code_and_drops_its_value() + { + var r = AtTimeEvaluator.Evaluate(At(25), A[2], A[3], Now, Now); + r.Value.ShouldBeNull(); // NOT 35 and NOT 30 + r.StatusCode.ShouldBe(0x80000000u); + r.SourceTimestampUtc.ShouldBe(At(25)); + } + + [Fact] + public void Exact_hit_returns_the_stored_value_and_status_unchanged() + { + var r = AtTimeEvaluator.Evaluate(At(20), A[2], A[3], At(21), Now); + r.Value.ShouldBe(30d); + r.StatusCode.ShouldBe(0x80000000u); + r.ServerTimestampUtc.ShouldBe(At(21)); + } + + [Fact] + public void No_previous_sample_is_BadNoData_with_no_value() + { + var r = AtTimeEvaluator.Evaluate(At(0).AddTicks(-1), null, A[0], Now, Now); + r.Value.ShouldBeNull(); + r.StatusCode.ShouldBe(0x809B0000u); + r.SourceTimestampUtc.ShouldBe(At(0).AddTicks(-1)); + } + + [Fact] + public void A_bad_next_neighbour_holds_the_previous_value_uncertain() + { + var r = AtTimeEvaluator.Evaluate(At(15), A[1], A[2], Now, Now); + r.Value.ShouldBe(20d); // NOT 25 + r.StatusCode.ShouldBe(UncertainInterpolated); + } + + [Fact] + public void Between_two_good_doubles_is_linearly_interpolated_good() + { + AtTimeEvaluator.Evaluate(At(5), A[0], A[1], Now, Now).Value.ShouldBe(15d); + AtTimeEvaluator.Evaluate(At(2.5), A[0], A[1], Now, Now).Value.ShouldBe(12.5d); + var r = AtTimeEvaluator.Evaluate(At(32.5), A[3], A[4], Now, Now); + r.Value.ShouldBe(42.5d); + r.StatusCode.ShouldBe(GoodInterpolated); + r.SourceTimestampUtc.ShouldBe(At(32.5)); + } + + [Fact] + public void An_uncertain_neighbour_makes_the_interpolation_uncertain() + { + var r = AtTimeEvaluator.Evaluate(At(5), S(0, HistorianValue.FromDouble(10), Uncertain), + S(10, HistorianValue.FromDouble(20), Good), Now, Now); + r.Value.ShouldBe(15d); + r.StatusCode.ShouldBe(UncertainInterpolated); + } + + [Fact] + public void Int64_is_interpolated_and_rounded_away_from_zero_keeping_its_type() + { + var p = S(0, HistorianValue.FromInt64(10), Good); + var n = S(10, HistorianValue.FromInt64(15), Good); + AtTimeEvaluator.Evaluate(At(5), p, n, Now, Now).Value.ShouldBe(13L); // 12.5 ⇒ 13 + AtTimeEvaluator.Evaluate(At(2), p, n, Now, Now).Value.ShouldBe(11L); + } + + [Fact] + public void After_the_last_sample_the_value_is_stepped_good_until_now_then_uncertain() + { + var now = At(42); + var held = AtTimeEvaluator.Evaluate(At(41), A[4], null, now, now); + held.Value.ShouldBe(50d); + held.StatusCode.ShouldBe(GoodInterpolated); + var future = AtTimeEvaluator.Evaluate(At(45), A[4], null, now, now); + future.Value.ShouldBe(50d); + future.StatusCode.ShouldBe(UncertainInterpolated); + } + + [Fact] + public void A_non_numeric_kind_is_stepped_with_the_previous_status_plus_interpolated() + { + var p = S(0, HistorianValue.FromString("a"), Uncertain); + var n = S(10, HistorianValue.FromString("b"), Good); + var r = AtTimeEvaluator.Evaluate(At(5), p, n, Now, Now); + r.Value.ShouldBe("a"); + r.StatusCode.ShouldBe(Uncertain | 0x402u); + } +} diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/BucketPlanTests.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/BucketPlanTests.cs new file mode 100644 index 00000000..71ed9889 --- /dev/null +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/BucketPlanTests.cs @@ -0,0 +1,71 @@ +using Shouldly; +using Xunit; +using ZB.MOM.WW.OtOpcUa.Core.Abstractions; +using static ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests.Fixture; + +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests; + +public sealed class BucketPlanTests +{ + [Fact] + public void BucketPlan_rejects_more_than_100k_buckets() + { + var ex = Should.Throw(() => + BucketPlan.Create(DateTime.UnixEpoch, DateTime.UnixEpoch.AddSeconds(100_001), TimeSpan.FromSeconds(1))); + ex.ParamName.ShouldBe("interval"); + } + + [Fact] + public void Exactly_100k_one_tick_buckets_are_allowed() + { + var p = BucketPlan.Create(At(0), At(0).AddTicks(100_000), TimeSpan.FromTicks(1)); + p.BucketCount.ShouldBe(100_000); + p.IntervalTicks.ShouldBe(1); + } + + [Fact] + public void A_partial_last_interval_is_its_own_bucket() + { + var p = BucketPlan.Create(At(0), At(50), TimeSpan.FromSeconds(20)); + p.BucketCount.ShouldBe(3); + p.BucketStart(2).ShouldBe(At(40)); + p.BucketStart(2).Kind.ShouldBe(DateTimeKind.Utc); + } + + [Fact] + public void Zero_or_oversized_interval_is_one_bucket_and_an_empty_window_none() + { + BucketPlan.Create(At(0), At(80), TimeSpan.Zero).BucketCount.ShouldBe(1); + var wide = BucketPlan.Create(At(0), At(80), TimeSpan.FromDays(1)); + wide.BucketCount.ShouldBe(1); + wide.IntervalTicks.ShouldBe(80 * TimeSpan.TicksPerSecond); + BucketPlan.Create(At(0), At(0), TimeSpan.FromSeconds(1)).BucketCount.ShouldBe(0); + } + + [Fact] + public void A_reverse_window_is_planned_forward_and_assembled_descending() + { + var p = BucketPlan.Create(At(80), At(0), TimeSpan.FromSeconds(20)); + p.Reversed.ShouldBeTrue(); + p.StartUtc.ShouldBe(At(0)); + p.EndUtc.ShouldBe(At(80)); + var s = p.Assemble(new Dictionary(), t => new DataValueSnapshot(null, 0, t, t)); + s.Select(x => x.SourceTimestampUtc).ShouldBe([At(60), At(40), At(20), At(0)]); + } + + [Fact] + public void Assemble_fills_missing_buckets_with_the_empty_factory() + { + var p = BucketPlan.Create(At(0), At(60), TimeSpan.FromSeconds(20)); + var hit = new DataValueSnapshot(7d, 0, At(20), At(20)); + var s = p.Assemble(new Dictionary { [1] = hit }, t => new DataValueSnapshot(null, 1u, t, t)); + s.Select(x => x.Value).ShouldBe([null, (object?)7d, null]); + s[0].SourceTimestampUtc.ShouldBe(At(0)); + s[2].SourceTimestampUtc.ShouldBe(At(40)); + } + + [Fact] + public void StartUnixTicks_is_ticks_since_the_unix_epoch() + => BucketPlan.Create(DateTime.UnixEpoch.AddTicks(1234567), DateTime.UnixEpoch.AddSeconds(10), TimeSpan.Zero) + .StartUnixTicks.ShouldBe(1234567L); +} diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/Fixture.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/Fixture.cs new file mode 100644 index 00000000..8c0a85f9 --- /dev/null +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/Fixture.cs @@ -0,0 +1,37 @@ +using ZB.MOM.WW.OtOpcUa.Core.Abstractions; + +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests; + +/// The SqlServer integration fixture's time base and tag A, as plain values (no database). +internal static class Fixture +{ + /// 2026-10-09 08:00:00.1234567 UTC — deliberately not millisecond-aligned (same as ReadTestDb.T0). + public static readonly DateTime T0 = new DateTime(2026, 10, 9, 8, 0, 0, DateTimeKind.Utc).AddTicks(1_234_567); + + public const uint Good = 0x00000000u; + public const uint Bad = 0x80000000u; + public const uint BadNoCommunication = 0x80310000u; + public const uint Uncertain = 0x40000000u; + + // Hand-computed literals, copied from the SqlServer integration tests. + public const uint GoodInterpolated = 0x00000402u; + public const uint UncertainInterpolated = 0x40A40402u; + public const uint GoodCalc = 0x00000401u; + public const uint UncertainCalc = 0x40A40401u; + public const uint NoDataCalc = 0x809B0401u; + public const uint InvalidInputsCalc = 0x80D60401u; + + public static DateTime At(double seconds) => T0.AddTicks((long)(seconds * TimeSpan.TicksPerSecond)); + + public static StoredSample S(double seconds, HistorianValue v, uint status) => new(At(seconds), v, status); + + /// Tag A: +0=10, +10=20, +20=Bad(30), +30=40, +40=50 (Double). + public static readonly StoredSample[] A = + [ + S(0, HistorianValue.FromDouble(10), Good), + S(10, HistorianValue.FromDouble(20), Good), + S(20, HistorianValue.FromDouble(30), Bad), + S(30, HistorianValue.FromDouble(40), Good), + S(40, HistorianValue.FromDouble(50), Good), + ]; +} diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/HistoryUpdatePlannerTests.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/HistoryUpdatePlannerTests.cs new file mode 100644 index 00000000..598c9a7c --- /dev/null +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/HistoryUpdatePlannerTests.cs @@ -0,0 +1,111 @@ +using Shouldly; +using Xunit; +using ZB.MOM.WW.OtOpcUa.Core.Abstractions; +using static ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.HistorianStatusCodes; + +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests; + +/// Direct tests of the HistoryUpdate result rules (literals from the SqlServer UpdateTests). +public sealed class HistoryUpdatePlannerTests +{ + private static readonly DateTime T0 = new DateTime(2026, 10, 9, 8, 0, 0, DateTimeKind.Utc).AddTicks(1234567); + + private static DateTime T(int s) => T0.AddSeconds(s); + + private static HistorizationValue V(DateTime? t, double v, uint status = 0) => new(t, HistorianValue.FromDouble(v), status); + + private static Dictionary Wrote(UpdatePlan p, UpsertOutcome o) => + p.Winners.ToDictionary(w => w.Row.TimeUtc.Ticks, _ => o); + + [Fact] + public void Insert_duplicates_keep_the_first_and_later_ones_are_BadEntryExists() + { + var plan = HistoryUpdatePlanner.Plan("t", HistoryUpdateMode.Insert, [V(T(1), 1), V(T(1), 2), V(T(2), 3)]); + plan.Winners.Select(w => w.Index).ShouldBe([0, 2]); + plan.Winners[0].Row.Value.Number.ShouldBe(1); + HistoryUpdatePlanner.Complete(plan, HistoryUpdateMode.Insert, Wrote(plan, UpsertOutcome.Inserted)) + .ShouldBe([GoodEntryInserted, BadEntryExists, GoodEntryInserted]); + } + + [Fact] + public void Replace_duplicates_last_wins_and_every_occurrence_shares_its_code() + { + var plan = HistoryUpdatePlanner.Plan("t", HistoryUpdateMode.Replace, [V(T(1), 1), V(T(2), 5), V(T(1), 2)]); + plan.Winners.Select(w => w.Index).ShouldBe([1, 2]); + var done = HistoryUpdatePlanner.Complete(plan, HistoryUpdateMode.Replace, + new Dictionary { [T(1).Ticks] = UpsertOutcome.Replaced }); + done.ShouldBe([GoodEntryReplaced, BadNoEntryExists, GoodEntryReplaced]); + } + + [Fact] + public void An_unwritten_winner_is_BadEntryExists_for_Insert_and_BadNoEntryExists_for_Replace() + { + var ins = HistoryUpdatePlanner.Plan("t", HistoryUpdateMode.Insert, [V(T(1), 1)]); + HistoryUpdatePlanner.Complete(ins, HistoryUpdateMode.Insert, new Dictionary()).ShouldBe([BadEntryExists]); + var rep = HistoryUpdatePlanner.Plan("t", HistoryUpdateMode.Replace, [V(T(1), 1)]); + HistoryUpdatePlanner.Complete(rep, HistoryUpdateMode.Replace, new Dictionary()).ShouldBe([BadNoEntryExists]); + } + + [Fact] + public void Invalid_items_are_refused_and_the_rest_proceed() + { + var plan = HistoryUpdatePlanner.Plan("t", HistoryUpdateMode.Insert, + [ + V(null, 1), V(DateTime.MinValue, 1), V(T(1), double.NaN), + new(T(2), HistorianValue.Empty, 0u), // Empty with a Good status + new(T(3), HistorianValue.Empty, 0x80310000u), // Empty with a Bad status: an outage marker, accepted + V(T(4), 4), + ]); + plan.Results.ShouldBe([BadInvalidTimestamp, BadInvalidTimestamp, BadInvalidArgument, BadInvalidArgument, null, null]); + plan.Winners.Select(w => w.Index).ShouldBe([4, 5]); + } + + [Fact] + public void A_tag_name_over_400_characters_refuses_every_item_and_sends_nothing() + { + var plan = HistoryUpdatePlanner.Plan(new string('x', 401), HistoryUpdateMode.Update, [V(T(1), 1), V(T(2), 2)]); + plan.Winners.ShouldBeEmpty(); + HistoryUpdatePlanner.Complete(plan, HistoryUpdateMode.Update, new Dictionary()) + .ShouldBe([BadInvalidArgument, BadInvalidArgument]); + HistoryUpdatePlanner.Plan(new string('x', 400), HistoryUpdateMode.Update, [V(T(1), 1)]).Winners.Count.ShouldBe(1); + } + + [Fact] + public void Resolution_10_truncates_before_dedup_and_the_superseded_duplicate_shares_the_winners_code() + { + // T0 ends in …567 ticks: +2 (…569) lands in the same microsecond (…560); +3 would cross into the next. + var plan = HistoryUpdatePlanner.Plan("t", HistoryUpdateMode.Update, [V(T0, 1), V(T0.AddTicks(2), 2)], resolutionTicks: 10); + plan.Winners.Count.ShouldBe(1); + plan.Winners[0].Index.ShouldBe(1); // last wins + plan.Winners[0].Row.TimeUtc.Ticks.ShouldBe(T0.Ticks - T0.Ticks % 10); // truncated + plan.Winners[0].Row.Value.Number.ShouldBe(2); + HistoryUpdatePlanner.Complete(plan, HistoryUpdateMode.Update, Wrote(plan, UpsertOutcome.Inserted)) + .ShouldBe([GoodEntryInserted, GoodEntryInserted]); + } + + [Fact] + public void Resolution_10_Insert_duplicates_after_truncation_are_BadEntryExists() + { + var plan = HistoryUpdatePlanner.Plan("t", HistoryUpdateMode.Insert, [V(T0, 1), V(T0.AddTicks(2), 2)], resolutionTicks: 10); + HistoryUpdatePlanner.Complete(plan, HistoryUpdateMode.Insert, Wrote(plan, UpsertOutcome.Inserted)) + .ShouldBe([GoodEntryInserted, BadEntryExists]); + } + + [Fact] + public void DeleteAtTime_first_occurrence_reports_the_outcome_and_minvalue_is_an_invalid_timestamp() + { + IReadOnlyList times = [T(1), T(1), DateTime.MinValue, T(2)]; + HistoryUpdatePlanner.DeleteAtTimeResults(times, new HashSet { T(1).Ticks }, tagKnown: true) + .ShouldBe([Good, BadNoEntryExists, BadInvalidTimestamp, BadNoEntryExists]); + HistoryUpdatePlanner.DeleteAtTimeResults(times, new HashSet { T(1).Ticks }, tagKnown: false) + .ShouldBe([BadNoEntryExists, BadNoEntryExists, BadInvalidTimestamp, BadNoEntryExists]); + } + + [Fact] + public void DeleteRaw_is_good_only_for_a_known_tag_with_deleted_rows() + { + HistoryUpdatePlanner.DeleteRawResult(tagKnown: true, 3).ShouldBe(Good); + HistoryUpdatePlanner.DeleteRawResult(tagKnown: true, 0).ShouldBe(BadNoData); + HistoryUpdatePlanner.DeleteRawResult(tagKnown: false, 5).ShouldBe(BadNoData); + } +} diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/LiveWritePreparerTests.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/LiveWritePreparerTests.cs new file mode 100644 index 00000000..258bfb61 --- /dev/null +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/LiveWritePreparerTests.cs @@ -0,0 +1,37 @@ +using Shouldly; +using Xunit; +using ZB.MOM.WW.OtOpcUa.Core.Abstractions; + +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests; + +public sealed class LiveWritePreparerTests +{ + private static readonly DateTime T = new DateTime(2026, 10, 9, 8, 0, 0, DateTimeKind.Utc).AddTicks(1234567); + + [Fact] + public void Prepare_truncates_to_microseconds_before_dedup_when_resolution_is_10() + { + // The brief's example used AddTicks(3), but …567 + 3 = …570 crosses into the NEXT microsecond; +2 (…569) stays + // in the same one (…560), which is the case the test means to pin. + var t = T; + var rows = LiveWritePreparer.Prepare([new(t, HistorianValue.FromDouble(1), 0), new(t.AddTicks(2), HistorianValue.FromDouble(2), 0)], () => t, 10); + rows.Count.ShouldBe(1); + rows[0].TimeUtc.Ticks.ShouldBe(t.Ticks - t.Ticks % 10); + rows[0].Value.Number.ShouldBe(1); + } + + [Fact] + public void Resolution_1_keeps_100ns_ticks_distinct() + { + var rows = LiveWritePreparer.Prepare([new(T, HistorianValue.FromDouble(1), 0), new(T.AddTicks(3), HistorianValue.FromDouble(2), 0)], () => T); + rows.Select(r => r.TimeUtc.Ticks).ShouldBe([T.Ticks, T.Ticks + 3]); + } + + [Fact] + public void Truncation_also_applies_to_the_clock_and_the_result_is_utc() + { + var rows = LiveWritePreparer.Prepare([new(null, HistorianValue.FromDouble(1), 0)], () => T, 10); + rows[0].TimeUtc.ShouldBe(new DateTime(T.Ticks - 7, DateTimeKind.Utc)); + rows[0].TimeUtc.Kind.ShouldBe(DateTimeKind.Utc); + } +} diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/SharedHelperTests.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/SharedHelperTests.cs new file mode 100644 index 00000000..bc15f86b --- /dev/null +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/SharedHelperTests.cs @@ -0,0 +1,67 @@ +using Shouldly; +using Xunit; +using ZB.MOM.WW.OtOpcUa.Core.Abstractions; + +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests; + +/// EventContinuation, HistorianHealthCounters, ClampSeverity and the StoredSample quality clamp. +public sealed class SharedHelperTests +{ + [Fact] + public void Event_continuation_is_ticks_pipe_event_id() + { + var t = new DateTime(2026, 10, 9, 8, 0, 0, DateTimeKind.Utc).AddTicks(1234567); + EventContinuation.Encode(t, "abc").ShouldBe($"{t.Ticks.ToString(System.Globalization.CultureInfo.InvariantCulture)}|abc"); + EventContinuation.Encode(new DateTime(42), "id").ShouldBe("42|id"); + } + + [Theory] + [InlineData(-5, 0)] + [InlineData(700, 700)] + [InlineData(70_000, 65_535)] + public void Event_severity_is_clamped_to_ushort(int stored, int expected) => + HistoricalEventRows.ClampSeverity(stored).ShouldBe((ushort)expected); + + [Fact] + public void Health_counters_open_only_after_a_success_with_no_failure_since() + { + var h = new HistorianHealthCounters(); + h.Snapshot().ProcessConnectionOpen.ShouldBeFalse(); + h.RecordSuccess(); + h.Snapshot().ProcessConnectionOpen.ShouldBeTrue(); + h.RecordFailure(new InvalidOperationException("boom")); + var s = h.Snapshot(); + s.ProcessConnectionOpen.ShouldBeFalse(); + s.EventConnectionOpen.ShouldBeFalse(); + s.ConsecutiveFailures.ShouldBe(1); + s.TotalQueries.ShouldBe(2); + s.LastError.ShouldBe("InvalidOperationException: boom"); + h.RecordSuccess(); + h.Snapshot().ConsecutiveFailures.ShouldBe(0); + h.Snapshot().ProcessConnectionOpen.ShouldBeTrue(); + } + + [Fact] + public void A_cancellation_is_not_a_health_failure() + { + var h = new HistorianHealthCounters(); + h.RecordFailure(new OperationCanceledException()); + h.RecordFailure(new TaskCanceledException()); + var s = h.Snapshot(); + s.TotalQueries.ShouldBe(0); + s.TotalFailures.ShouldBe(0); + s.LastError.ShouldBeNull(); + } + + [Fact] + public void A_stored_empty_sample_with_a_non_bad_status_is_served_BadNoData() + { + var t = DateTime.UnixEpoch; + StoredSample.FromStored(t, HistorianValue.Empty, 0u).Status.ShouldBe(0x809B0000u); + StoredSample.FromStored(t, HistorianValue.Empty, 0x40000000u).Status.ShouldBe(0x809B0000u); + StoredSample.FromStored(t, HistorianValue.Empty, 0x80310000u).Status.ShouldBe(0x80310000u); + StoredSample.FromStored(t, HistorianValue.FromDouble(1), 0x40000000u).Status.ShouldBe(0x40000000u); + StoredSample.FromStored(t, HistorianValue.Empty, 0x80310000u).IsUsable.ShouldBeFalse(); + StoredSample.FromStored(t, HistorianValue.FromDouble(1), 0x40000000u).IsGood.ShouldBeFalse(); + } +} From 082e808d0dd0bb49f172bfff13f58b58470e630f Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Sun, 11 Oct 2026 01:42:22 -0400 Subject: [PATCH 08/23] test(historian): pin Uncertain-input degradation (T1 review) Two Sql.Common tests close review finding I-1: an Uncertain NEXT neighbour makes an AtTime interpolation Uncertain, and an Uncertain (non-Bad) sample in a bucket makes the aggregate Uncertain. Each falsified with the reviewer's surviving mutation. Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe --- .../AggregateEvaluatorTests.cs | 10 ++++++++++ .../AtTimeEvaluatorTests.cs | 9 +++++++++ 2 files changed, 19 insertions(+) diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/AggregateEvaluatorTests.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/AggregateEvaluatorTests.cs index 4e9270e5..72a4a54b 100644 --- a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/AggregateEvaluatorTests.cs +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/AggregateEvaluatorTests.cs @@ -119,4 +119,14 @@ public sealed class AggregateEvaluatorTests AggregateEvaluator.Evaluate(HistoryAggregateType.TimeAverage, RowsA()[0], At(0), Now)); ex.Aggregate.ShouldBe(HistoryAggregateType.TimeAverage); } + + [Fact] + public void An_uncertain_sample_in_a_bucket_degrades_the_aggregate_to_uncertain() + { + // Two usable samples, one Good and one Uncertain, no Bad: the value is computed, the status is Uncertain. + var row = new BucketRow { Total = 2, Usable = 2, Good = 1, Bad = 0, Avg = 15, Min = 10, Max = 20, Sum = 30 }; + var avg = AggregateEvaluator.Evaluate(HistoryAggregateType.Average, row, At(0), Now); + avg.Value.ShouldBe(15d); + avg.StatusCode.ShouldBe(UncertainCalc); + } } diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/AtTimeEvaluatorTests.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/AtTimeEvaluatorTests.cs index 5ab37704..70dc5a96 100644 --- a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/AtTimeEvaluatorTests.cs +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/AtTimeEvaluatorTests.cs @@ -106,4 +106,13 @@ public sealed class AtTimeEvaluatorTests r.Value.ShouldBe("a"); r.StatusCode.ShouldBe(Uncertain | 0x402u); } + + [Fact] + public void An_uncertain_next_neighbour_makes_the_interpolation_uncertain() + { + var r = AtTimeEvaluator.Evaluate(At(5), S(0, HistorianValue.FromDouble(10), Good), + S(10, HistorianValue.FromDouble(20), Uncertain), Now, Now); + r.Value.ShouldBe(15d); + r.StatusCode.ShouldBe(UncertainInterpolated); + } } From c5b33aa12309e9c6f65ae7b551d90e328fae86ab Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Sun, 11 Oct 2026 01:50:51 -0400 Subject: [PATCH 09/23] test(historian): backend conformance suite (shared source), run against SQL Server Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe --- .../AtTimeConformance.cs} | 114 ++++----- .../Historian.Conformance/ConformanceData.cs | 40 ++++ .../IConformanceBackend.cs | 89 +++++++ .../LiveWriteConformance.cs | 97 ++++++++ .../ProcessedConformance.cs} | 183 +++++++-------- .../QualityContractConformance.cs} | 55 ++--- .../RawReadConformance.cs | 134 +++++++++++ .../TagNameConformance.cs | 64 +++++ .../UpdateConformance.cs} | 124 ++++------ .../LongNameTests.cs | 48 ---- .../RawReadTests.cs | 218 ------------------ .../ReadTestDb.cs | 87 +++++++ .../SqlServerConformance.cs | 55 +++++ .../SqlServerConformanceBackend.cs | 77 +++++++ .../WriteTests.cs | 95 -------- ...istorian.SqlServer.IntegrationTests.csproj | 6 + 16 files changed, 847 insertions(+), 639 deletions(-) rename tests/Drivers/{ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/AtTimeReadTests.cs => Historian.Conformance/AtTimeConformance.cs} (59%) create mode 100644 tests/Drivers/Historian.Conformance/ConformanceData.cs create mode 100644 tests/Drivers/Historian.Conformance/IConformanceBackend.cs create mode 100644 tests/Drivers/Historian.Conformance/LiveWriteConformance.cs rename tests/Drivers/{ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/ProcessedReadTests.cs => Historian.Conformance/ProcessedConformance.cs} (66%) rename tests/Drivers/{ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/QualityContractTests.cs => Historian.Conformance/QualityContractConformance.cs} (57%) create mode 100644 tests/Drivers/Historian.Conformance/RawReadConformance.cs create mode 100644 tests/Drivers/Historian.Conformance/TagNameConformance.cs rename tests/Drivers/{ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/UpdateTests.cs => Historian.Conformance/UpdateConformance.cs} (62%) create mode 100644 tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/ReadTestDb.cs create mode 100644 tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/SqlServerConformance.cs create mode 100644 tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/SqlServerConformanceBackend.cs diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/AtTimeReadTests.cs b/tests/Drivers/Historian.Conformance/AtTimeConformance.cs similarity index 59% rename from tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/AtTimeReadTests.cs rename to tests/Drivers/Historian.Conformance/AtTimeConformance.cs index 110beb80..8a2acea7 100644 --- a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/AtTimeReadTests.cs +++ b/tests/Drivers/Historian.Conformance/AtTimeConformance.cs @@ -2,18 +2,16 @@ using Shouldly; using Xunit; using ZB.MOM.WW.OtOpcUa.Core.Abstractions; -namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests; +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Conformance; /// /// AtTime rules (spec §4, as amended after the T10 review): exact, linear, stepped, no-prior, and the two /// Bad-neighbour cases — inside an outage (Bad previous) ⇒ null with the outage's own Bad code; before an /// outage (usable previous, Bad next) ⇒ previous value held, Uncertain. Never bridges. /// -[Collection(SqlHistorianCollection.Name)] -public sealed class AtTimeReadTests(SqlHistorianFixture fx) +public abstract class AtTimeConformance : ConformanceBase { private static readonly CancellationToken Ct = CancellationToken.None; - private static DateTime At(double s) => ReadTestDb.At(s); // Hand-computed status literals: Interpolated = InfoType DataValue 0x400 | Interpolated 0x2. private const uint GoodInterpolated = 0x00000402u; @@ -33,13 +31,12 @@ public sealed class AtTimeReadTests(SqlHistorianFixture fx) [Fact] public async Task Exact_hit_returns_the_stored_value_and_status() { - var db = await ReadTestDb.CreateAsync(fx); - await db.SeedAAsync(); - var s = (await db.Source().ReadAtTimeAsync("A", [At(10), At(20)], Ct)).Samples; + await ConformanceData.SeedAAsync(B); + var s = (await B.CreateDataSource().ReadAtTimeAsync("A", [At(10), At(20)], Ct)).Samples; s[0].Value.ShouldBe(20d); s[0].StatusCode.ShouldBe(0u); - s[0].SourceTimestampUtc.ShouldBe(At(10)); - s[0].ServerTimestampUtc.ShouldBe(At(11)); // the stored ServerTimeUtc + s[0].SourceTimestampUtc.ShouldBe(B.Round(At(10))); + s[0].ServerTimestampUtc.ShouldBe(B.Round(At(11))); // the stored ServerTimeUtc s[1].Value.ShouldBe(30d); // a stored Bad sample comes back as stored s[1].StatusCode.ShouldBe(0x80000000u); } @@ -47,33 +44,31 @@ public sealed class AtTimeReadTests(SqlHistorianFixture fx) [Fact] public async Task Between_two_good_doubles_is_linearly_interpolated_good() { - var db = await ReadTestDb.CreateAsync(fx); - await db.SeedAAsync(); - var s = (await db.Source().ReadAtTimeAsync("A", [At(5), At(2.5), At(32.5)], Ct)).Samples; + await ConformanceData.SeedAAsync(B); + var s = (await B.CreateDataSource().ReadAtTimeAsync("A", [At(5), At(2.5), At(32.5)], Ct)).Samples; s[0].Value.ShouldBe(15d); // 10 + (20-10) * 5/10 s[1].Value.ShouldBe(12.5d); // 10 + 10 * 2.5/10 s[2].Value.ShouldBe(42.5d); // 40 + 10 * 2.5/10 s.ShouldAllBe(x => x.StatusCode == GoodInterpolated); - s[0].SourceTimestampUtc.ShouldBe(At(5)); // stamped at the requested (sub-ms) time + s[0].SourceTimestampUtc.ShouldBe(B.Round(At(5))); // stamped at the requested (sub-ms) time } [Fact] public async Task No_previous_sample_is_BadNoData() { - var db = await ReadTestDb.CreateAsync(fx); - await db.SeedAAsync(); - var s = (await db.Source().ReadAtTimeAsync("A", [At(0).AddTicks(-1)], Ct)).Samples.Single(); + await ConformanceData.SeedAAsync(B); + var before = At(0).AddTicks(-B.ResolutionTicks); + var s = (await B.CreateDataSource().ReadAtTimeAsync("A", [before], Ct)).Samples.Single(); s.Value.ShouldBeNull(); s.StatusCode.ShouldBe(BadNoData); - s.SourceTimestampUtc.ShouldBe(At(0).AddTicks(-1)); + s.SourceTimestampUtc.ShouldBe(B.Round(before)); } [Fact] public async Task A_bad_next_neighbour_holds_the_previous_value_uncertain() { - var db = await ReadTestDb.CreateAsync(fx); - await db.SeedAAsync(); - var s = (await db.Source().ReadAtTimeAsync("A", [At(15)], Ct)).Samples.Single(); + await ConformanceData.SeedAAsync(B); + var s = (await B.CreateDataSource().ReadAtTimeAsync("A", [At(15)], Ct)).Samples.Single(); s.Value.ShouldBe(20d); // NOT 25 (interpolating toward the Bad 30) s.StatusCode.ShouldBe(UncertainInterpolated); } @@ -81,23 +76,21 @@ public sealed class AtTimeReadTests(SqlHistorianFixture fx) [Fact] public async Task A_bad_previous_neighbour_is_inside_an_outage_and_keeps_its_bad_code() { - var db = await ReadTestDb.CreateAsync(fx); - await db.SeedAAsync(); - var s = (await db.Source().ReadAtTimeAsync("A", [At(25)], Ct)).Samples.Single(); + await ConformanceData.SeedAAsync(B); + var s = (await B.CreateDataSource().ReadAtTimeAsync("A", [At(25)], Ct)).Samples.Single(); s.Value.ShouldBeNull(); // NOT 35 (30→40) and NOT 30 (a Bad value) s.StatusCode.ShouldBe(0x80000000u); // the previous sample's Bad code — never upgraded to Uncertain - s.SourceTimestampUtc.ShouldBe(At(25)); + s.SourceTimestampUtc.ShouldBe(B.Round(At(25))); } [Fact] public async Task Interpolation_never_bridges_an_outage_marker() { - var db = await ReadTestDb.CreateAsync(fx); - await db.SeedAsync("M", - (At(0), HistorianValue.FromDouble(10), ReadTestDb.Good), - (At(5), HistorianValue.Empty, ReadTestDb.BadNoCommunication), - (At(10), HistorianValue.FromDouble(30), ReadTestDb.Good)); - var s = (await db.Source().ReadAtTimeAsync("M", [At(2.5), At(7.5), At(5)], Ct)).Samples; + await B.SeedAsync("M", + (At(0), HistorianValue.FromDouble(10), Good), + (At(5), HistorianValue.Empty, BadNoCommunication), + (At(10), HistorianValue.FromDouble(30), Good)); + var s = (await B.CreateDataSource().ReadAtTimeAsync("M", [At(2.5), At(7.5), At(5)], Ct)).Samples; s[0].Value.ShouldBe(10d); // bridging 10→30 would give 15 s[0].StatusCode.ShouldBe(UncertainInterpolated); s[1].Value.ShouldBeNull(); // bridging would give 25 @@ -109,9 +102,8 @@ public sealed class AtTimeReadTests(SqlHistorianFixture fx) [Fact] public async Task After_the_last_sample_the_value_is_stepped() { - var db = await ReadTestDb.CreateAsync(fx); - await db.SeedAAsync(); - var s = (await db.Source().ReadAtTimeAsync("A", [At(45)], Ct)).Samples.Single(); + await ConformanceData.SeedAAsync(B); + var s = (await B.CreateDataSource().ReadAtTimeAsync("A", [At(45)], Ct)).Samples.Single(); s.Value.ShouldBe(50d); s.StatusCode.ShouldBe(GoodInterpolated); } @@ -119,9 +111,8 @@ public sealed class AtTimeReadTests(SqlHistorianFixture fx) [Fact] public async Task A_time_after_now_stepped_from_the_last_sample_is_uncertain() { - var db = await ReadTestDb.CreateAsync(fx); - await db.SeedAAsync(); // last sample +40 = 50, Good - var src = db.Source(new FixedClock(new DateTimeOffset(At(42)))); + await ConformanceData.SeedAAsync(B); // last sample +40 = 50, Good + var src = B.CreateDataSource(new ConformanceClock(new DateTimeOffset(At(42)))); var s = (await src.ReadAtTimeAsync("A", [At(41), At(45)], Ct)).Samples; s[0].Value.ShouldBe(50d); s[0].StatusCode.ShouldBe(GoodInterpolated); // after the last sample but not after now: held Good @@ -132,11 +123,10 @@ public sealed class AtTimeReadTests(SqlHistorianFixture fx) [Fact] public async Task An_uncertain_neighbour_makes_the_interpolation_uncertain() { - var db = await ReadTestDb.CreateAsync(fx); - await db.SeedAsync("U", - (At(0), HistorianValue.FromDouble(10), ReadTestDb.Uncertain), - (At(10), HistorianValue.FromDouble(20), ReadTestDb.Good)); - var s = (await db.Source().ReadAtTimeAsync("U", [At(5)], Ct)).Samples.Single(); + await B.SeedAsync("U", + (At(0), HistorianValue.FromDouble(10), Uncertain), + (At(10), HistorianValue.FromDouble(20), Good)); + var s = (await B.CreateDataSource().ReadAtTimeAsync("U", [At(5)], Ct)).Samples.Single(); s.Value.ShouldBe(15d); s.StatusCode.ShouldBe(UncertainInterpolated); } @@ -144,11 +134,10 @@ public sealed class AtTimeReadTests(SqlHistorianFixture fx) [Fact] public async Task Int64_is_interpolated_and_rounded_keeping_its_type() { - var db = await ReadTestDb.CreateAsync(fx); - await db.SeedAsync("I", - (At(0), HistorianValue.FromInt64(10), ReadTestDb.Good), - (At(10), HistorianValue.FromInt64(15), ReadTestDb.Good)); - var s = (await db.Source().ReadAtTimeAsync("I", [At(5), At(2)], Ct)).Samples; + await B.SeedAsync("I", + (At(0), HistorianValue.FromInt64(10), Good), + (At(10), HistorianValue.FromInt64(15), Good)); + var s = (await B.CreateDataSource().ReadAtTimeAsync("I", [At(5), At(2)], Ct)).Samples; s[0].Value.ShouldBe(13L); // 12.5 ⇒ 13 (away from zero) s[1].Value.ShouldBe(11L); // 11.0 s[0].StatusCode.ShouldBe(GoodInterpolated); @@ -157,18 +146,17 @@ public sealed class AtTimeReadTests(SqlHistorianFixture fx) [Fact] public async Task Boolean_string_and_datetime_are_stepped_from_the_previous_sample() { - var db = await ReadTestDb.CreateAsync(fx); - var d0 = new DateTime(2020, 1, 1, 0, 0, 0, DateTimeKind.Utc).AddTicks(3); - await db.SeedAsync("B", - (At(0), HistorianValue.FromBoolean(true), ReadTestDb.Good), - (At(10), HistorianValue.FromBoolean(false), ReadTestDb.Good)); - await db.SeedAsync("S", - (At(0), HistorianValue.FromString("alpha"), ReadTestDb.Uncertain), - (At(10), HistorianValue.FromString("beta"), ReadTestDb.Good)); - await db.SeedAsync("D", - (At(0), HistorianValue.FromDateTime(d0), ReadTestDb.Good), - (At(10), HistorianValue.FromDateTime(d0.AddDays(1)), ReadTestDb.Good)); - var src = db.Source(); + var d0 = new DateTime(2020, 1, 1, 0, 0, 0, DateTimeKind.Utc).AddTicks(3 * B.ResolutionTicks); + await B.SeedAsync("B", + (At(0), HistorianValue.FromBoolean(true), Good), + (At(10), HistorianValue.FromBoolean(false), Good)); + await B.SeedAsync("S", + (At(0), HistorianValue.FromString("alpha"), Uncertain), + (At(10), HistorianValue.FromString("beta"), Good)); + await B.SeedAsync("D", + (At(0), HistorianValue.FromDateTime(d0), Good), + (At(10), HistorianValue.FromDateTime(d0.AddDays(1)), Good)); + var src = B.CreateDataSource(); var b = (await src.ReadAtTimeAsync("B", [At(9)], Ct)).Samples.Single(); b.Value.ShouldBe(true); b.StatusCode.ShouldBe(GoodInterpolated); @@ -181,19 +169,17 @@ public sealed class AtTimeReadTests(SqlHistorianFixture fx) [Fact] public async Task Result_matches_request_length_and_order_including_duplicates() { - var db = await ReadTestDb.CreateAsync(fx); - await db.SeedAAsync(); - var s = (await db.Source().ReadAtTimeAsync("A", [At(5), At(-1), At(5), At(10)], Ct)).Samples; + await ConformanceData.SeedAAsync(B); + var s = (await B.CreateDataSource().ReadAtTimeAsync("A", [At(5), At(-1), At(5), At(10)], Ct)).Samples; s.Count.ShouldBe(4); s.Select(x => x.Value).ShouldBe([15d, null, 15d, 20d]); - s.Select(x => x.SourceTimestampUtc).ShouldBe([At(5), At(-1), At(5), At(10)]); + s.Select(x => x.SourceTimestampUtc).ShouldBe([B.Round(At(5)), B.Round(At(-1)), B.Round(At(5)), B.Round(At(10))]); } [Fact] public async Task Empty_request_and_unknown_tag() { - var db = await ReadTestDb.CreateAsync(fx); - var src = db.Source(); + var src = B.CreateDataSource(); (await src.ReadAtTimeAsync("A", [], Ct)).Samples.ShouldBeEmpty(); var s = (await src.ReadAtTimeAsync("nope", [At(1), At(2)], Ct)).Samples; s.Count.ShouldBe(2); diff --git a/tests/Drivers/Historian.Conformance/ConformanceData.cs b/tests/Drivers/Historian.Conformance/ConformanceData.cs new file mode 100644 index 00000000..ac08471d --- /dev/null +++ b/tests/Drivers/Historian.Conformance/ConformanceData.cs @@ -0,0 +1,40 @@ +using ZB.MOM.WW.OtOpcUa.Core.Abstractions; + +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Conformance; + +/// Shared fixture data of the conformance suite. is deliberately NOT millisecond- or microsecond-aligned. +public static class ConformanceData +{ + /// Base time: 2026-10-09 08:00:00.1234567 UTC (sub-millisecond ticks on purpose). + public static readonly DateTime T0 = new DateTime(2026, 10, 9, 8, 0, 0, DateTimeKind.Utc).AddTicks(1_234_567); + + /// Good. + public const uint Good = 0x00000000u; + /// Bad (generic). + public const uint Bad = 0x80000000u; + /// BadNoCommunication. + public const uint BadNoCommunication = 0x80310000u; + /// Uncertain (generic). + public const uint Uncertain = 0x40000000u; + + /// T0 + seconds. + public static DateTime At(double seconds) => T0.AddTicks((long)(seconds * TimeSpan.TicksPerSecond)); + + /// The fixture tag A: +0=10, +10=20, +20=Bad(30), +30=40, +40=50 (Double). + public static Task SeedAAsync(IConformanceBackend b, string name = "A") => b.SeedAsync(name, + (At(0), HistorianValue.FromDouble(10), Good), + (At(10), HistorianValue.FromDouble(20), Good), + (At(20), HistorianValue.FromDouble(30), Bad), + (At(30), HistorianValue.FromDouble(40), Good), + (At(40), HistorianValue.FromDouble(50), Good)); +} + +/// A settable clock for a data source's "now". +public sealed class ConformanceClock(DateTimeOffset now) : TimeProvider +{ + /// The current instant. + public DateTimeOffset Now { get; set; } = now; + + /// + public override DateTimeOffset GetUtcNow() => Now; +} diff --git a/tests/Drivers/Historian.Conformance/IConformanceBackend.cs b/tests/Drivers/Historian.Conformance/IConformanceBackend.cs new file mode 100644 index 00000000..55077224 --- /dev/null +++ b/tests/Drivers/Historian.Conformance/IConformanceBackend.cs @@ -0,0 +1,89 @@ +using Xunit; +using ZB.MOM.WW.OtOpcUa.Core.Abstractions; + +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Conformance; + +/// +/// One historian backend under test. The conformance suite talks to a backend only through this contract +/// (plus Core.Abstractions), so the same hand-computed expectations run against SQL Server and QuestDB. +/// Each instance owns an isolated store (a fresh schema / table set) so tests never see each other's data. +/// +public interface IConformanceBackend : IAsyncDisposable +{ + /// Storage timestamp resolution in .NET ticks: 1 for SQL Server (100 ns), 10 for QuestDB (µs). + long ResolutionTicks { get; } + + /// Truncates a timestamp to the backend's storage resolution (UTC kind). Every timestamp assertion goes through it. + /// The timestamp. + /// The value the backend stores/serves for . + DateTime Round(DateTime t) => new(t.Ticks - t.Ticks % ResolutionTicks, DateTimeKind.Utc); + + /// A fresh data source (fresh catalog cache) over this backend's store. + /// Its "now"; null means the system clock. + IHistorianDataSource CreateDataSource(TimeProvider? clock = null); + + /// A fresh live-value writer over this backend's store. + /// The clock used to stamp values that carry no timestamp; null means the system clock. + IHistorianValueWriter CreateValueWriter(Func? utcNow = null); + + /// A fresh history updater over this backend's store. + IHistorianDataUpdater CreateUpdater(); + + /// + /// Writes rows directly in the backend's schema (never through the writer or updater), status verbatim, creating the + /// tag if needed. The stored server timestamp of each row is its source time + 1 s, so tests can tell them apart. + /// + /// The tag name. + /// The rows to store. + Task SeedAsync(string tag, params (DateTime T, HistorianValue V, uint Status)[] samples); + + /// + /// Live (non-deleted) rows for a tag, ascending by time, read directly from storage. The status is returned + /// VERBATIM as stored — there is no read-side clamp (an Empty row stored non-Bad reads back non-Bad here). + /// + /// The tag name. + Task> ReadBackAsync(string tag); + + /// Whether a tag with exactly this name exists in storage (no truncation of long names). + /// The tag name. + Task TagExistsAsync(string tag); + + /// The data type the backend recorded for a tag, or null when the tag does not exist. + /// The tag name. + Task TagDataTypeAsync(string tag); +} + +/// +/// Base of every conformance class. A backend's test project subclasses each abstract suite class once, +/// supplying , which must call its fixture's skip gate so the tests report Skipped +/// (not Failed) when the backend is unavailable. +/// +public abstract class ConformanceBase : IAsyncLifetime +{ + /// Good. + protected const uint Good = ConformanceData.Good; + /// Bad (generic). + protected const uint Bad = ConformanceData.Bad; + /// BadNoCommunication. + protected const uint BadNoCommunication = ConformanceData.BadNoCommunication; + /// Uncertain (generic). + protected const uint Uncertain = ConformanceData.Uncertain; + + /// The backend under test for the current test. + protected IConformanceBackend B { get; private set; } = null!; + + /// Creates this test's isolated backend; calls the fixture's skip gate first. + protected abstract Task CreateBackendAsync(); + + /// T0 + seconds. + protected static DateTime At(double s) => ConformanceData.At(s); + + /// + public async ValueTask InitializeAsync() => B = await CreateBackendAsync(); + + /// + public async ValueTask DisposeAsync() + { + if (B is not null) await B.DisposeAsync(); + } +} diff --git a/tests/Drivers/Historian.Conformance/LiveWriteConformance.cs b/tests/Drivers/Historian.Conformance/LiveWriteConformance.cs new file mode 100644 index 00000000..c6960e19 --- /dev/null +++ b/tests/Drivers/Historian.Conformance/LiveWriteConformance.cs @@ -0,0 +1,97 @@ +using Shouldly; +using Xunit; +using ZB.MOM.WW.OtOpcUa.Core.Abstractions; +using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common; + +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Conformance; + +/// Live value writer behaviour: round-trips, idempotent redelivery, first-wins duplicates, non-finite doubles. +public abstract class LiveWriteConformance : ConformanceBase +{ + // Non-ms-aligned: sub-millisecond ticks must round-trip exactly (to the backend's resolution). + private static readonly DateTime T0 = new DateTime(2026, 10, 9, 1, 2, 3, DateTimeKind.Utc).AddTicks(1234567); + + [Fact] + public async Task Every_kind_round_trips_with_exact_ticks_and_status() + { + var w = B.CreateValueWriter(); + var r = B.ResolutionTicks; + var dt = new DateTime(2026, 1, 2, 3, 4, 5, DateTimeKind.Utc).AddTicks(77 * r); + HistorizationValue[] vals = + [ + new(T0, HistorianValue.FromDouble(1.25), 0), + new(T0.AddTicks(1 * r), HistorianValue.FromInt64(long.MaxValue), 0x40000000u), + new(T0.AddTicks(2 * r), HistorianValue.FromBoolean(true), 0), + new(T0.AddTicks(3 * r), HistorianValue.FromString("héllo"), 0), + new(T0.AddTicks(4 * r), HistorianValue.FromDateTime(dt), 0), + new(T0.AddTicks(5 * r), HistorianValue.Empty, 0x80310000u), + ]; + (await w.WriteLiveValuesAsync("tag.a", vals, CancellationToken.None)).ShouldBeTrue(); + var rows = await B.ReadBackAsync("tag.a"); + rows.Count.ShouldBe(6); + for (var i = 0; i < 6; i++) rows[i].T.Ticks.ShouldBe(B.Round(vals[i].TimestampUtc!.Value).Ticks); + rows.Select(x => x.V).ToList().ShouldBe(vals.Select(v => v.Value).ToList()); + rows[1].Status.ShouldBe(0x40000000u); + rows[5].Status.ShouldBe(0x80310000u); + } + + [Fact] + public async Task Writing_the_same_batch_twice_is_idempotent() + { + var w = B.CreateValueWriter(); + HistorizationValue[] vals = [new(T0, HistorianValue.FromDouble(1), 0), new(T0.AddSeconds(1), HistorianValue.FromDouble(2), 0)]; + (await w.WriteLiveValuesAsync("tag.b", vals, CancellationToken.None)).ShouldBeTrue(); + (await w.WriteLiveValuesAsync("tag.b", vals, CancellationToken.None)).ShouldBeTrue(); + // Overlapping batch: only the new row lands, the stored first value is untouched. + (await w.WriteLiveValuesAsync("tag.b", [new(T0, HistorianValue.FromDouble(99), 0), new(T0.AddSeconds(2), HistorianValue.FromDouble(3), 0)], CancellationToken.None)).ShouldBeTrue(); + var rows = await B.ReadBackAsync("tag.b"); + rows.Count.ShouldBe(3); + rows[0].V.Number.ShouldBe(1); + } + + [Fact] + public async Task Within_batch_duplicate_timestamps_do_not_fail_and_first_wins() + { + var w = B.CreateValueWriter(); + (await w.WriteLiveValuesAsync("tag.c", [new(T0, HistorianValue.FromDouble(1), 0), new(T0, HistorianValue.FromDouble(2), 0)], CancellationToken.None)).ShouldBeTrue(); + var rows = await B.ReadBackAsync("tag.c"); + rows.Count.ShouldBe(1); + rows[0].V.Number.ShouldBe(1); + } + + [Fact] + public async Task Non_finite_doubles_store_as_Empty_forced_Bad_and_do_not_fail_the_write() + { + var w = B.CreateValueWriter(); + var r = B.ResolutionTicks; + (await w.WriteLiveValuesAsync("tag.d", [ + new(T0, HistorianValue.FromDouble(double.NaN), 0x40000000u), + new(T0.AddTicks(1 * r), HistorianValue.FromDouble(double.PositiveInfinity), 0), + new(T0.AddTicks(2 * r), HistorianValue.FromDouble(7), 0)], CancellationToken.None)).ShouldBeTrue(); + var rows = await B.ReadBackAsync("tag.d"); + rows.Count.ShouldBe(3); + rows[0].V.Kind.ShouldBe(HistorianValueKind.Empty); + rows[0].Status.ShouldBe(HistorianStatusCodes.BadOutOfRange); // Uncertain NaN ⇒ Bad (I-1) + rows[1].Status.ShouldBe(HistorianStatusCodes.BadOutOfRange); // Good +Inf ⇒ Bad (I-1) + rows[2].V.Number.ShouldBe(7); + } + + [Fact] + public async Task Empty_batch_succeeds_without_a_call_and_null_timestamp_is_stamped() + { + var w = B.CreateValueWriter(); + (await w.WriteLiveValuesAsync("tag.e", [], CancellationToken.None)).ShouldBeTrue(); + (await w.WriteLiveValuesAsync("tag.e", [new(null, HistorianValue.FromInt64(5), 0)], CancellationToken.None)).ShouldBeTrue(); + var rows = await B.ReadBackAsync("tag.e"); + rows.Count.ShouldBe(1); + (DateTime.UtcNow - rows[0].T).ShouldBeLessThan(TimeSpan.FromMinutes(1)); + } + + [Fact] + public async Task Tag_is_created_with_the_first_typed_values_data_type() + { + var w = B.CreateValueWriter(); + await w.WriteLiveValuesAsync("tag.f", [new(T0, HistorianValue.Empty, 0x80310000u), new(T0.AddTicks(B.ResolutionTicks), HistorianValue.FromString("x"), 0)], CancellationToken.None); + (await B.TagDataTypeAsync("tag.f")).ShouldBe(DriverDataType.String); + } +} diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/ProcessedReadTests.cs b/tests/Drivers/Historian.Conformance/ProcessedConformance.cs similarity index 66% rename from tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/ProcessedReadTests.cs rename to tests/Drivers/Historian.Conformance/ProcessedConformance.cs index bc98a74c..cd579922 100644 --- a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/ProcessedReadTests.cs +++ b/tests/Drivers/Historian.Conformance/ProcessedConformance.cs @@ -2,20 +2,18 @@ using Shouldly; using Xunit; using ZB.MOM.WW.OtOpcUa.Core.Abstractions; -namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests; +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Conformance; /// /// Push-down aggregates over tag A (+0=10, +10=20, +20=Bad(30), +30=40, +40=50), 20 s intervals from the -/// sub-millisecond , window [t0, t0+80s) ⇒ four buckets: +/// sub-millisecond , window [t0, t0+80s) ⇒ four buckets: /// B0 [0,20) = {10, 20}; B1 [20,40) = {Bad 30, 40}; B2 [40,60) = {50}; B3 [60,80) = {} (empty). /// Every expected value below is hand-computed. /// -[Collection(SqlHistorianCollection.Name)] -public sealed class ProcessedReadTests(SqlHistorianFixture fx) +public abstract class ProcessedConformance : ConformanceBase { private static readonly CancellationToken Ct = CancellationToken.None; private static readonly TimeSpan Twenty = TimeSpan.FromSeconds(20); - private static DateTime At(double s) => ReadTestDb.At(s); // Calculated = InfoType DataValue 0x400 | Calculated 0x1, OR-ed onto the code. private const uint GoodCalc = 0x00000401u; @@ -23,8 +21,8 @@ public sealed class ProcessedReadTests(SqlHistorianFixture fx) private const uint NoDataCalc = 0x809B0401u; // BadNoData | 0x401 private const uint InvalidInputsCalc = 0x80D60401u; // BadAggregateInvalidInputs | 0x401 - private static async Task ReadA(ReadTestDb db, HistoryAggregateType agg) => - await db.Source().ReadProcessedAsync("A", At(0), At(80), Twenty, agg, Ct); + private async Task ReadA(HistoryAggregateType agg) => + await B.CreateDataSource().ReadProcessedAsync("A", At(0), At(80), Twenty, agg, Ct); [Fact] public void The_status_literals_carry_the_SDK_Calculated_aggregate_bits() @@ -48,23 +46,21 @@ public sealed class ProcessedReadTests(SqlHistorianFixture fx) [InlineData(HistoryAggregateType.Range, 10d, 0d, 0d)] public async Task Numeric_aggregates_over_the_fixture(HistoryAggregateType agg, double b0, double b1, double b2) { - var db = await ReadTestDb.CreateAsync(fx); - await db.SeedAAsync(); - var s = (await ReadA(db, agg)).Samples; + await ConformanceData.SeedAAsync(B); + var s = (await ReadA(agg)).Samples; s.Count.ShouldBe(4); s.Select(x => x.Value).ShouldBe([b0, b1, b2, null]); // B1 holds a Bad sample alongside a usable one ⇒ Uncertain; the empty B3 ⇒ BadNoData. s.Select(x => x.StatusCode).ShouldBe([GoodCalc, UncertainCalc, GoodCalc, NoDataCalc]); - s.Select(x => x.SourceTimestampUtc).ShouldBe([At(0), At(20), At(40), At(60)]); // bucket starts + s.Select(x => x.SourceTimestampUtc).ShouldBe([B.Round(At(0)), B.Round(At(20)), B.Round(At(40)), B.Round(At(60))]); // bucket starts } [Fact] public async Task Count_counts_usable_samples_as_int() { - var db = await ReadTestDb.CreateAsync(fx); - await db.SeedAAsync(); - var s = (await ReadA(db, HistoryAggregateType.Count)).Samples; + await ConformanceData.SeedAAsync(B); + var s = (await ReadA(HistoryAggregateType.Count)).Samples; s.Select(x => x.Value).ShouldBe([(object?)2, 1, 1, null]); s.Select(x => x.StatusCode).ShouldBe([GoodCalc, UncertainCalc, GoodCalc, NoDataCalc]); } @@ -74,9 +70,8 @@ public sealed class ProcessedReadTests(SqlHistorianFixture fx) [InlineData(HistoryAggregateType.PercentBad, 0d, 50d, 0d)] public async Task Percent_aggregates_count_every_sample(HistoryAggregateType agg, double b0, double b1, double b2) { - var db = await ReadTestDb.CreateAsync(fx); - await db.SeedAAsync(); - var s = (await ReadA(db, agg)).Samples; + await ConformanceData.SeedAAsync(B); + var s = (await ReadA(agg)).Samples; s.Select(x => x.Value).ShouldBe([b0, b1, b2, null]); // A percentage is a statement about quality, so it is not itself degraded by the Bad input. s.Select(x => x.StatusCode).ShouldBe([GoodCalc, GoodCalc, GoodCalc, NoDataCalc]); @@ -85,34 +80,32 @@ public sealed class ProcessedReadTests(SqlHistorianFixture fx) [Fact] public async Task Actual_time_aggregates_stamp_the_sample_time() { - var db = await ReadTestDb.CreateAsync(fx); - await db.SeedAAsync(); - var min = (await ReadA(db, HistoryAggregateType.MinimumActualTime)).Samples; + await ConformanceData.SeedAAsync(B); + var min = (await ReadA(HistoryAggregateType.MinimumActualTime)).Samples; min.Select(x => x.Value).ShouldBe([10d, 40d, 50d, null]); - min.Select(x => x.SourceTimestampUtc).ShouldBe([At(0), At(30), At(40), At(60)]); + min.Select(x => x.SourceTimestampUtc).ShouldBe([B.Round(At(0)), B.Round(At(30)), B.Round(At(40)), B.Round(At(60))]); min.Select(x => x.StatusCode).ShouldBe([GoodCalc, UncertainCalc, GoodCalc, NoDataCalc]); - var max = (await ReadA(db, HistoryAggregateType.MaximumActualTime)).Samples; + var max = (await ReadA(HistoryAggregateType.MaximumActualTime)).Samples; max.Select(x => x.Value).ShouldBe([20d, 40d, 50d, null]); - max.Select(x => x.SourceTimestampUtc).ShouldBe([At(10), At(30), At(40), At(60)]); + max.Select(x => x.SourceTimestampUtc).ShouldBe([B.Round(At(10)), B.Round(At(30)), B.Round(At(40)), B.Round(At(60))]); } [Fact] public async Task Actual_time_ties_resolve_to_the_earliest_sample() { - var db = await ReadTestDb.CreateAsync(fx); - await db.SeedAsync("T", - (At(0), HistorianValue.FromDouble(5), ReadTestDb.Good), - (At(2), HistorianValue.FromDouble(3), ReadTestDb.Good), - (At(4), HistorianValue.FromDouble(3), ReadTestDb.Good), - (At(6), HistorianValue.FromDouble(7), ReadTestDb.Good), - (At(8), HistorianValue.FromDouble(7), ReadTestDb.Good)); - var src = db.Source(); + await B.SeedAsync("T", + (At(0), HistorianValue.FromDouble(5), Good), + (At(2), HistorianValue.FromDouble(3), Good), + (At(4), HistorianValue.FromDouble(3), Good), + (At(6), HistorianValue.FromDouble(7), Good), + (At(8), HistorianValue.FromDouble(7), Good)); + var src = B.CreateDataSource(); var min = (await src.ReadProcessedAsync("T", At(0), At(10), TimeSpan.Zero, HistoryAggregateType.MinimumActualTime, Ct)).Samples.Single(); min.Value.ShouldBe(3d); - min.SourceTimestampUtc.ShouldBe(At(2)); + min.SourceTimestampUtc.ShouldBe(B.Round(At(2))); var max = (await src.ReadProcessedAsync("T", At(0), At(10), TimeSpan.Zero, HistoryAggregateType.MaximumActualTime, Ct)).Samples.Single(); max.Value.ShouldBe(7d); - max.SourceTimestampUtc.ShouldBe(At(6)); + max.SourceTimestampUtc.ShouldBe(B.Round(At(6))); } [Fact] @@ -120,16 +113,15 @@ public sealed class ProcessedReadTests(SqlHistorianFixture fx) { // T0 is not millisecond-aligned: a DATEDIFF(millisecond) bucket formula would push the sample one tick // before the +20 s edge into bucket 1 (it counts millisecond boundaries crossed, not elapsed time). - var db = await ReadTestDb.CreateAsync(fx); var edge = At(20); - await db.SeedAsync("E", - (At(0).AddTicks(-1), HistorianValue.FromDouble(1000), ReadTestDb.Good), // before start: excluded - (At(0), HistorianValue.FromDouble(1), ReadTestDb.Good), // == start: B0 - (edge.AddTicks(-1), HistorianValue.FromDouble(2), ReadTestDb.Good), // last tick of B0 - (edge, HistorianValue.FromDouble(4), ReadTestDb.Good), // first tick of B1 - (At(40).AddTicks(-1), HistorianValue.FromDouble(8), ReadTestDb.Good), // last tick of B1 - (At(40), HistorianValue.FromDouble(2000), ReadTestDb.Good)); // == end: excluded - var s = (await db.Source().ReadProcessedAsync("E", At(0), At(40), Twenty, HistoryAggregateType.Total, Ct)).Samples; + await B.SeedAsync("E", + (At(0).AddTicks(-B.ResolutionTicks), HistorianValue.FromDouble(1000), Good), // before start: excluded + (At(0), HistorianValue.FromDouble(1), Good), // == start: B0 + (edge.AddTicks(-B.ResolutionTicks), HistorianValue.FromDouble(2), Good), // last tick of B0 + (edge, HistorianValue.FromDouble(4), Good), // first tick of B1 + (At(40).AddTicks(-B.ResolutionTicks), HistorianValue.FromDouble(8), Good), // last tick of B1 + (At(40), HistorianValue.FromDouble(2000), Good)); // == end: excluded + var s = (await B.CreateDataSource().ReadProcessedAsync("E", At(0), At(40), Twenty, HistoryAggregateType.Total, Ct)).Samples; s.Select(x => x.Value).ShouldBe([3d, 12d]); s.ShouldAllBe(x => x.StatusCode == GoodCalc); } @@ -137,23 +129,21 @@ public sealed class ProcessedReadTests(SqlHistorianFixture fx) [Fact] public async Task A_partial_last_interval_is_truncated_at_end() { - var db = await ReadTestDb.CreateAsync(fx); - await db.SeedAAsync(); - var s = (await db.Source().ReadProcessedAsync("A", At(0), At(50), Twenty, HistoryAggregateType.Average, Ct)).Samples; - s.Select(x => x.SourceTimestampUtc).ShouldBe([At(0), At(20), At(40)]); + await ConformanceData.SeedAAsync(B); + var s = (await B.CreateDataSource().ReadProcessedAsync("A", At(0), At(50), Twenty, HistoryAggregateType.Average, Ct)).Samples; + s.Select(x => x.SourceTimestampUtc).ShouldBe([B.Round(At(0)), B.Round(At(20)), B.Round(At(40))]); s.Select(x => x.Value).ShouldBe([15d, 40d, 50d]); } [Fact] public async Task A_non_positive_interval_is_one_interval_over_the_window() { - var db = await ReadTestDb.CreateAsync(fx); - await db.SeedAAsync(); - var src = db.Source(); + await ConformanceData.SeedAAsync(B); + var src = B.CreateDataSource(); var avg = (await src.ReadProcessedAsync("A", At(0), At(50), TimeSpan.Zero, HistoryAggregateType.Average, Ct)).Samples.Single(); avg.Value.ShouldBe(30d); // usable {10, 20, 40, 50} avg.StatusCode.ShouldBe(UncertainCalc); - avg.SourceTimestampUtc.ShouldBe(At(0)); + avg.SourceTimestampUtc.ShouldBe(B.Round(At(0))); (await src.ReadProcessedAsync("A", At(0), At(50), TimeSpan.FromSeconds(-1), HistoryAggregateType.Count, Ct)) .Samples.Single().Value.ShouldBe(4); } @@ -163,11 +153,10 @@ public sealed class ProcessedReadTests(SqlHistorianFixture fx) { // 1601-01-01 is the OPC UA minimum DateTime. A nanosecond DATEDIFF_BIG overflows past ≈292 years (SQL 535), // and a ns-scaled interval would wrap a long; both must be gone. - var db = await ReadTestDb.CreateAsync(fx); - await db.SeedAAsync(); + await ConformanceData.SeedAAsync(B); var start = new DateTime(1601, 1, 1, 0, 0, 0, DateTimeKind.Utc); var end = new DateTime(9999, 12, 31, 0, 0, 0, DateTimeKind.Utc); - var src = db.Source(); + var src = B.CreateDataSource(); var avg = (await src.ReadProcessedAsync("A", start, end, TimeSpan.Zero, HistoryAggregateType.Average, Ct)).Samples.Single(); avg.Value.ShouldBe(30d); // usable {10, 20, 40, 50} avg.StatusCode.ShouldBe(UncertainCalc); @@ -189,11 +178,10 @@ public sealed class ProcessedReadTests(SqlHistorianFixture fx) // Hand-computed (calendar arithmetic, independent of SQL): 1601-01-01 → 2027-01-01 is 155 593 days = // 22 227.57 weeks ⇒ 22 228 intervals (under the 100k cap). 2026-10-09 08:00 is 155 509 days in ⇒ week // index 22 215, which starts Monday 2026-10-05 00:00; every fixture sample (t0 … t0+40 s) is in it. - var db = await ReadTestDb.CreateAsync(fx); - await db.SeedAAsync(); + await ConformanceData.SeedAAsync(B); var start = new DateTime(1601, 1, 1, 0, 0, 0, DateTimeKind.Utc); var end = new DateTime(2027, 1, 1, 0, 0, 0, DateTimeKind.Utc); - var s = (await db.Source().ReadProcessedAsync("A", start, end, TimeSpan.FromDays(7), HistoryAggregateType.Average, Ct)).Samples; + var s = (await B.CreateDataSource().ReadProcessedAsync("A", start, end, TimeSpan.FromDays(7), HistoryAggregateType.Average, Ct)).Samples; s.Count.ShouldBe(22_228); var hit = s[22_215]; @@ -207,31 +195,28 @@ public sealed class ProcessedReadTests(SqlHistorianFixture fx) [Fact] public async Task A_reverse_window_is_computed_forward_and_returned_descending() { - var db = await ReadTestDb.CreateAsync(fx); - await db.SeedAAsync(); - var s = (await db.Source().ReadProcessedAsync("A", At(80), At(0), Twenty, HistoryAggregateType.Average, Ct)).Samples; - s.Select(x => x.SourceTimestampUtc).ShouldBe([At(60), At(40), At(20), At(0)]); + await ConformanceData.SeedAAsync(B); + var s = (await B.CreateDataSource().ReadProcessedAsync("A", At(80), At(0), Twenty, HistoryAggregateType.Average, Ct)).Samples; + s.Select(x => x.SourceTimestampUtc).ShouldBe([B.Round(At(60)), B.Round(At(40)), B.Round(At(20)), B.Round(At(0))]); s.Select(x => x.Value).ShouldBe([null, 50d, 40d, 15d]); } [Fact] public async Task Equal_start_and_end_yields_no_intervals() { - var db = await ReadTestDb.CreateAsync(fx); - await db.SeedAAsync(); - (await db.Source().ReadProcessedAsync("A", At(10), At(10), Twenty, HistoryAggregateType.Average, Ct)) + await ConformanceData.SeedAAsync(B); + (await B.CreateDataSource().ReadProcessedAsync("A", At(10), At(10), Twenty, HistoryAggregateType.Average, Ct)) .Samples.ShouldBeEmpty(); } [Fact] public async Task Outage_markers_are_bad_samples_in_aggregates() { - var db = await ReadTestDb.CreateAsync(fx); - await db.SeedAsync("M", - (At(0), HistorianValue.FromDouble(10), ReadTestDb.Good), - (At(5), HistorianValue.Empty, ReadTestDb.BadNoCommunication), - (At(10), HistorianValue.FromDouble(30), ReadTestDb.Good)); - var src = db.Source(); + await B.SeedAsync("M", + (At(0), HistorianValue.FromDouble(10), Good), + (At(5), HistorianValue.Empty, BadNoCommunication), + (At(10), HistorianValue.FromDouble(30), Good)); + var src = B.CreateDataSource(); async Task One(HistoryAggregateType a) => (await src.ReadProcessedAsync("M", At(0), At(20), Twenty, a, Ct)).Samples.Single(); @@ -246,11 +231,10 @@ public sealed class ProcessedReadTests(SqlHistorianFixture fx) [Fact] public async Task An_interval_of_only_bad_samples_is_BadNoData_except_the_percentages() { - var db = await ReadTestDb.CreateAsync(fx); - await db.SeedAsync("Z", - (At(1), HistorianValue.Empty, ReadTestDb.BadNoCommunication), - (At(2), HistorianValue.FromDouble(5), ReadTestDb.Bad)); - var src = db.Source(); + await B.SeedAsync("Z", + (At(1), HistorianValue.Empty, BadNoCommunication), + (At(2), HistorianValue.FromDouble(5), Bad)); + var src = B.CreateDataSource(); async Task One(HistoryAggregateType a) => (await src.ReadProcessedAsync("Z", At(0), At(20), Twenty, a, Ct)).Samples.Single(); @@ -264,11 +248,10 @@ public sealed class ProcessedReadTests(SqlHistorianFixture fx) [Fact] public async Task Uncertain_samples_are_used_but_make_the_interval_uncertain() { - var db = await ReadTestDb.CreateAsync(fx); - await db.SeedAsync("U", - (At(1), HistorianValue.FromDouble(4), ReadTestDb.Uncertain), - (At(2), HistorianValue.FromDouble(6), ReadTestDb.Good)); - var s = (await db.Source().ReadProcessedAsync("U", At(0), At(20), Twenty, HistoryAggregateType.Average, Ct)).Samples.Single(); + await B.SeedAsync("U", + (At(1), HistorianValue.FromDouble(4), Uncertain), + (At(2), HistorianValue.FromDouble(6), Good)); + var s = (await B.CreateDataSource().ReadProcessedAsync("U", At(0), At(20), Twenty, HistoryAggregateType.Average, Ct)).Samples.Single(); s.Value.ShouldBe(5d); s.StatusCode.ShouldBe(UncertainCalc); } @@ -276,11 +259,10 @@ public sealed class ProcessedReadTests(SqlHistorianFixture fx) [Fact] public async Task String_tags_serve_count_start_end_and_refuse_numeric_aggregates() { - var db = await ReadTestDb.CreateAsync(fx); - await db.SeedAsync("S", - (At(0), HistorianValue.FromString("a"), ReadTestDb.Good), - (At(10), HistorianValue.FromString("b"), ReadTestDb.Good)); - var src = db.Source(); + await B.SeedAsync("S", + (At(0), HistorianValue.FromString("a"), Good), + (At(10), HistorianValue.FromString("b"), Good)); + var src = B.CreateDataSource(); async Task One(HistoryAggregateType a) => (await src.ReadProcessedAsync("S", At(0), At(20), Twenty, a, Ct)).Samples.Single(); @@ -304,22 +286,20 @@ public sealed class ProcessedReadTests(SqlHistorianFixture fx) [Fact] public async Task DateTime_tags_refuse_numeric_aggregates() { - var db = await ReadTestDb.CreateAsync(fx); - await db.SeedAsync("D", (At(0), HistorianValue.FromDateTime(At(-100)), ReadTestDb.Good)); - (await db.Source().ReadProcessedAsync("D", At(0), At(20), Twenty, HistoryAggregateType.Average, Ct)) + await B.SeedAsync("D", (At(0), HistorianValue.FromDateTime(At(-100)), Good)); + (await B.CreateDataSource().ReadProcessedAsync("D", At(0), At(20), Twenty, HistoryAggregateType.Average, Ct)) .Samples.Single().StatusCode.ShouldBe(InvalidInputsCalc); } [Fact] public async Task Boolean_tags_aggregate_as_zero_or_one() { - var db = await ReadTestDb.CreateAsync(fx); - await db.SeedAsync("B", - (At(0), HistorianValue.FromBoolean(true), ReadTestDb.Good), - (At(5), HistorianValue.FromBoolean(false), ReadTestDb.Good), - (At(10), HistorianValue.FromBoolean(true), ReadTestDb.Good), - (At(15), HistorianValue.FromBoolean(true), ReadTestDb.Good)); - var src = db.Source(); + await B.SeedAsync("B", + (At(0), HistorianValue.FromBoolean(true), Good), + (At(5), HistorianValue.FromBoolean(false), Good), + (At(10), HistorianValue.FromBoolean(true), Good), + (At(15), HistorianValue.FromBoolean(true), Good)); + var src = B.CreateDataSource(); async Task One(HistoryAggregateType a) => (await src.ReadProcessedAsync("B", At(0), At(20), Twenty, a, Ct)).Samples.Single(); (await One(HistoryAggregateType.Average)).Value.ShouldBe(0.75d); @@ -331,8 +311,7 @@ public sealed class ProcessedReadTests(SqlHistorianFixture fx) [Fact] public async Task Unknown_tag_yields_every_interval_as_BadNoData() { - var db = await ReadTestDb.CreateAsync(fx); - var s = (await db.Source().ReadProcessedAsync("nope", At(0), At(60), Twenty, HistoryAggregateType.Average, Ct)).Samples; + var s = (await B.CreateDataSource().ReadProcessedAsync("nope", At(0), At(60), Twenty, HistoryAggregateType.Average, Ct)).Samples; s.Count.ShouldBe(3); s.ShouldAllBe(x => x.StatusCode == NoDataCalc && x.Value == null); } @@ -352,8 +331,7 @@ public sealed class ProcessedReadTests(SqlHistorianFixture fx) [InlineData(HistoryAggregateType.VariancePopulation)] public async Task Unsupported_aggregates_throw_not_supported_and_are_not_health_failures(HistoryAggregateType agg) { - var db = await ReadTestDb.CreateAsync(fx); - var src = db.Source(); + var src = B.CreateDataSource(); var ex = await Should.ThrowAsync( () => src.ReadProcessedAsync("A", At(0), At(20), Twenty, agg, Ct)); ex.Aggregate.ShouldBe(agg); @@ -363,13 +341,12 @@ public sealed class ProcessedReadTests(SqlHistorianFixture fx) [Fact] public async Task The_bucket_count_is_capped_at_100000() { - var db = await ReadTestDb.CreateAsync(fx); - var src = db.Source(); - var oneTick = TimeSpan.FromTicks(1); - (await src.ReadProcessedAsync("nope", At(0), At(0).AddTicks(100_000), oneTick, HistoryAggregateType.Count, Ct)) + var src = B.CreateDataSource(); + var oneTick = TimeSpan.FromTicks(B.ResolutionTicks); + (await src.ReadProcessedAsync("nope", At(0), At(0).AddTicks(100_000 * B.ResolutionTicks), oneTick, HistoryAggregateType.Count, Ct)) .Samples.Count.ShouldBe(100_000); await Should.ThrowAsync( - () => src.ReadProcessedAsync("nope", At(0), At(0).AddTicks(100_001), oneTick, HistoryAggregateType.Count, Ct)); + () => src.ReadProcessedAsync("nope", At(0), At(0).AddTicks(100_001 * B.ResolutionTicks), oneTick, HistoryAggregateType.Count, Ct)); var h = src.GetHealthSnapshot(); h.TotalFailures.ShouldBe(0); // a client-request error, not a backend fault } diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/QualityContractTests.cs b/tests/Drivers/Historian.Conformance/QualityContractConformance.cs similarity index 57% rename from tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/QualityContractTests.cs rename to tests/Drivers/Historian.Conformance/QualityContractConformance.cs index c08a856a..39a42d1c 100644 --- a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/QualityContractTests.cs +++ b/tests/Drivers/Historian.Conformance/QualityContractConformance.cs @@ -3,51 +3,41 @@ using Xunit; using ZB.MOM.WW.OtOpcUa.Core.Abstractions; using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common; -namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests; +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Conformance; /// /// Final-review I-1: a stored sample is never served with a better quality than "no value". An Empty (null-value) /// sample is always Bad — forced Bad by the value writer, refused non-Bad by the updater, and clamped to /// BadNoData on read for rows written before either rule existed. /// -[Collection(SqlHistorianCollection.Name)] -public sealed class QualityContractTests(SqlHistorianFixture fx) +public abstract class QualityContractConformance : ConformanceBase { private static readonly CancellationToken Ct = CancellationToken.None; - private static DateTime At(double s) => ReadTestDb.At(s); private const uint BadOutOfRange = 0x803C0000u; private const uint BadNoData = 0x809B0000u; private const uint BadInvalidArgument = 0x80AB0000u; - private const uint BadNoCommunication = 0x80310000u; private static bool IsBad(uint status) => (status & 0x80000000u) != 0; - private static SqlServerHistorianValueWriter Writer(ReadTestDb db) => - new(db.Conn, db.Boot, new SqlServerTagCatalog(db.Conn, db.Boot)); - - private static SqlServerHistorianDataUpdater Updater(ReadTestDb db) => - new(db.Conn, new SqlServerTagCatalog(db.Conn, db.Boot)); - [Fact] public async Task A_good_NaN_written_through_the_real_writer_reads_back_Bad_on_Raw_and_AtTime() { - var db = await ReadTestDb.CreateAsync(fx); - (await Writer(db).WriteLiveValuesAsync("nan", [ + (await B.CreateValueWriter().WriteLiveValuesAsync("nan", [ new(At(0), HistorianValue.FromDouble(10), 0u), new(At(10), HistorianValue.FromDouble(double.NaN), 0u), new(At(20), HistorianValue.FromDouble(double.PositiveInfinity), 0x40000000u), new(At(30), HistorianValue.FromDouble(double.NegativeInfinity), BadNoCommunication), ], Ct)).ShouldBeTrue(); - var raw = (await db.Source().ReadRawAsync("nan", At(0), At(100), 0, Ct)).Samples; + var raw = (await B.CreateDataSource().ReadRawAsync("nan", At(0), At(100), 0, Ct)).Samples; raw.Count.ShouldBe(4); raw[1].Value.ShouldBeNull(); raw[1].StatusCode.ShouldBe(BadOutOfRange); // Good NaN ⇒ stored Bad raw[2].StatusCode.ShouldBe(BadOutOfRange); // Uncertain +Inf ⇒ stored Bad raw[3].StatusCode.ShouldBe(BadNoCommunication); // an existing Bad code is kept - var at = (await db.Source().ReadAtTimeAsync("nan", [At(10), At(15)], Ct)).Samples; + var at = (await B.CreateDataSource().ReadAtTimeAsync("nan", [At(10), At(15)], Ct)).Samples; at.ShouldAllBe(s => s.Value == null); at.ShouldAllBe(s => IsBad(s.StatusCode)); at[1].StatusCode.ShouldBe(BadOutOfRange); // inside the "outage" the NaN opened @@ -56,42 +46,41 @@ public sealed class QualityContractTests(SqlHistorianFixture fx) [Fact] public async Task A_non_bad_Empty_written_through_the_real_writer_is_stored_BadNoData() { - var db = await ReadTestDb.CreateAsync(fx); - (await Writer(db).WriteLiveValuesAsync("empty", [ + (await B.CreateValueWriter().WriteLiveValuesAsync("empty", [ new(At(0), HistorianValue.Empty, 0u), new(At(1), HistorianValue.Empty, 0x40000000u), new(At(2), HistorianValue.Empty, BadNoCommunication), ], Ct)).ShouldBeTrue(); - // Read the stored column directly so the read-side clamp cannot mask a writer that forgot to force Bad. - (await db.ScalarAsync($"SELECT COUNT(*) FROM {db.Conn.Q("Sample")} WHERE StatusCode >= 0")).ShouldBe(0); - var raw = (await db.Source().ReadRawAsync("empty", At(0), At(10), 0, Ct)).Samples; + // Read the stored status directly so the read-side clamp cannot mask a writer that forgot to force Bad. + var stored = await B.ReadBackAsync("empty"); + stored.Count.ShouldBe(3); + stored.ShouldAllBe(r => IsBad(r.Status)); + var raw = (await B.CreateDataSource().ReadRawAsync("empty", At(0), At(10), 0, Ct)).Samples; raw.Select(s => s.StatusCode).ShouldBe([BadNoData, BadNoData, BadNoCommunication]); } [Fact] public async Task A_legacy_good_Empty_row_is_clamped_to_BadNoData_on_Raw_and_AtTime() { - var db = await ReadTestDb.CreateAsync(fx); - // Seeded with plain INSERTs, bypassing the writer: what a pre-fix build could have stored. - await db.SeedAsync("legacy", - (At(0), HistorianValue.FromDouble(10), ReadTestDb.Good), - (At(10), HistorianValue.Empty, ReadTestDb.Good), - (At(20), HistorianValue.Empty, ReadTestDb.Uncertain), - (At(30), HistorianValue.FromDouble(40), ReadTestDb.Good)); + // Seeded directly, bypassing the writer: what a pre-fix build could have stored. + await B.SeedAsync("legacy", + (At(0), HistorianValue.FromDouble(10), Good), + (At(10), HistorianValue.Empty, Good), + (At(20), HistorianValue.Empty, Uncertain), + (At(30), HistorianValue.FromDouble(40), Good)); - var raw = (await db.Source().ReadRawAsync("legacy", At(0), At(100), 0, Ct)).Samples; - raw.Select(s => s.StatusCode).ShouldBe([ReadTestDb.Good, BadNoData, BadNoData, ReadTestDb.Good]); + var raw = (await B.CreateDataSource().ReadRawAsync("legacy", At(0), At(100), 0, Ct)).Samples; + raw.Select(s => s.StatusCode).ShouldBe([Good, BadNoData, BadNoData, Good]); - var at = (await db.Source().ReadAtTimeAsync("legacy", [At(10), At(15), At(25)], Ct)).Samples; + var at = (await B.CreateDataSource().ReadAtTimeAsync("legacy", [At(10), At(15), At(25)], Ct)).Samples; at.ShouldAllBe(s => s.Value == null && s.StatusCode == BadNoData); } [Fact] public async Task The_updater_refuses_a_non_bad_Empty_and_accepts_a_bad_one() { - var db = await ReadTestDb.CreateAsync(fx); - var r = await Updater(db).UpdateDataAsync("u", HistoryUpdateMode.Insert, [ + var r = await B.CreateUpdater().UpdateDataAsync("u", HistoryUpdateMode.Insert, [ new(At(0), HistorianValue.Empty, 0u), new(At(1), HistorianValue.Empty, 0x40000000u), new(At(2), HistorianValue.Empty, BadNoCommunication), @@ -99,7 +88,7 @@ public sealed class QualityContractTests(SqlHistorianFixture fx) ], Ct); r.ShouldBe([BadInvalidArgument, BadInvalidArgument, HistorianStatusCodes.GoodEntryInserted, HistorianStatusCodes.GoodEntryInserted]); - (await db.Source().ReadRawAsync("u", At(0), At(10), 0, Ct)).Samples + (await B.CreateDataSource().ReadRawAsync("u", At(0), At(10), 0, Ct)).Samples .Select(s => s.StatusCode).ShouldBe([BadNoCommunication, 0u]); } } diff --git a/tests/Drivers/Historian.Conformance/RawReadConformance.cs b/tests/Drivers/Historian.Conformance/RawReadConformance.cs new file mode 100644 index 00000000..07380824 --- /dev/null +++ b/tests/Drivers/Historian.Conformance/RawReadConformance.cs @@ -0,0 +1,134 @@ +using Shouldly; +using Xunit; +using ZB.MOM.WW.OtOpcUa.Core.Abstractions; + +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Conformance; + +/// Raw-read behaviour over the fixture tag A (see ). +public abstract class RawReadConformance : ConformanceBase +{ + private static readonly CancellationToken Ct = CancellationToken.None; + + private static double?[] Values(HistoryReadResult r) => r.Samples.Select(s => (double?)s.Value).ToArray(); + + [Fact] + public async Task Forward_read_is_start_inclusive_end_exclusive_ascending_with_exact_ticks() + { + await ConformanceData.SeedAAsync(B); + var r = await B.CreateDataSource().ReadRawAsync("A", At(10), At(40), 0, Ct); + + Values(r).ShouldBe([20d, 30d, 40d]); // +10 included, +40 (== end) excluded + r.Samples.Select(s => s.SourceTimestampUtc!.Value.Ticks).ShouldBe([B.Round(At(10)).Ticks, B.Round(At(20)).Ticks, B.Round(At(30)).Ticks]); + r.Samples.Select(s => s.StatusCode).ShouldBe([Good, Bad, Good]); + r.Samples[0].SourceTimestampUtc!.Value.Kind.ShouldBe(DateTimeKind.Utc); + r.Samples[0].ServerTimestampUtc.Ticks.ShouldBe(B.Round(At(11)).Ticks); // stored ServerTimeUtc, not "now" + r.ContinuationPoint.ShouldBeNull(); + } + + [Fact] + public async Task Forward_read_honours_the_limit() + { + await ConformanceData.SeedAAsync(B); + Values(await B.CreateDataSource().ReadRawAsync("A", At(0), At(100), 2, Ct)).ShouldBe([10d, 20d]); + } + + [Fact] + public async Task Zero_max_values_is_unbounded() + { + await ConformanceData.SeedAAsync(B); + Values(await B.CreateDataSource().ReadRawAsync("A", At(-1), At(100), 0, Ct)).ShouldBe([10d, 20d, 30d, 40d, 50d]); + Values(await B.CreateDataSource().ReadRawAsync("A", At(-1), At(100), uint.MaxValue, Ct)).Length.ShouldBe(5); + } + + [Fact] + public async Task Reverse_read_is_descending_over_end_exclusive_start_inclusive() + { + await ConformanceData.SeedAAsync(B); + // start=+30, end=+10 ⇒ (+10, +30] descending. + Values(await B.CreateDataSource().ReadRawAsync("A", At(30), At(10), 0, Ct)).ShouldBe([40d, 30d]); + Values(await B.CreateDataSource().ReadRawAsync("A", At(100), At(-1), 2, Ct)).ShouldBe([50d, 40d]); + } + + [Fact] + public async Task Equal_start_and_end_is_a_point_read() + { + await ConformanceData.SeedAAsync(B); + Values(await B.CreateDataSource().ReadRawAsync("A", At(10), At(10), 0, Ct)).ShouldBe([20d]); + // One resolution unit later: the stored sample is no longer at that instant. + var later = At(10).AddTicks(B.ResolutionTicks); + (await B.CreateDataSource().ReadRawAsync("A", later, later, 0, Ct)).Samples.ShouldBeEmpty(); + } + + [Fact] + public async Task A_resumed_page_from_the_last_timestamp_inclusive_continues_without_gaps() + { + // The node manager's paging: read cap N, resume at the last returned timestamp INCLUSIVE, trim it. + await ConformanceData.SeedAAsync(B); + var src = B.CreateDataSource(); + var p1 = await src.ReadRawAsync("A", At(0), At(100), 2, Ct); + var p2 = await src.ReadRawAsync("A", p1.Samples[^1].SourceTimestampUtc!.Value, At(100), 2, Ct); + Values(p1).ShouldBe([10d, 20d]); + Values(p2).ShouldBe([20d, 30d]); // first is the boundary sample the node manager trims + } + + [Fact] + public async Task Unknown_tag_is_an_empty_result_and_reads_never_create_tags() + { + var src = B.CreateDataSource(); + (await src.ReadRawAsync("nope", At(0), At(10), 0, Ct)).Samples.ShouldBeEmpty(); + await src.ReadAtTimeAsync("nope", [At(1)], Ct); + await src.ReadProcessedAsync("nope", At(0), At(10), TimeSpan.FromSeconds(5), HistoryAggregateType.Average, Ct); + (await B.TagExistsAsync("nope")).ShouldBeFalse(); + src.GetHealthSnapshot().TotalFailures.ShouldBe(0); + } + + [Fact] + public async Task Outage_markers_and_bad_samples_come_back_with_their_bad_status() + { + await B.SeedAsync("M", + (At(0), HistorianValue.FromDouble(10), Good), + (At(5), HistorianValue.Empty, BadNoCommunication), + (At(7), HistorianValue.FromDouble(99), Bad)); + var r = await B.CreateDataSource().ReadRawAsync("M", At(0), At(10), 0, Ct); + r.Samples.Count.ShouldBe(3); + r.Samples[1].Value.ShouldBeNull(); + r.Samples[1].StatusCode.ShouldBe(0x80310000u); + r.Samples[2].Value.ShouldBe(99d); + r.Samples[2].StatusCode.ShouldBe(0x80000000u); + } + + [Fact] + public async Task Every_kind_round_trips_to_its_CLR_type() + { + var dt = new DateTime(2025, 1, 2, 3, 4, 5, DateTimeKind.Utc).AddTicks(7 * B.ResolutionTicks); + await B.SeedAsync("K", + (At(0), HistorianValue.FromInt64(-9_000_000_000L), Good), + (At(1), HistorianValue.FromBoolean(true), Good), + (At(2), HistorianValue.FromString("héllo"), Good), + (At(3), HistorianValue.FromDateTime(dt), Good), + (At(4), HistorianValue.FromDouble(1.5), Uncertain)); + var s = (await B.CreateDataSource().ReadRawAsync("K", At(0), At(5), 0, Ct)).Samples; + s[0].Value.ShouldBe(-9_000_000_000L); + s[1].Value.ShouldBe(true); + s[2].Value.ShouldBe("héllo"); + s[3].Value.ShouldBe(dt); + ((DateTime)s[3].Value!).Kind.ShouldBe(DateTimeKind.Utc); + s[4].Value.ShouldBe(1.5); + s[4].StatusCode.ShouldBe(0x40000000u); + } + + [Fact] + public async Task Successful_reads_are_counted_in_health() + { + await ConformanceData.SeedAAsync(B); + var src = B.CreateDataSource(); + await src.ReadRawAsync("A", At(0), At(10), 0, Ct); + await src.ReadRawAsync("A", At(0), At(10), 0, Ct); + var h = src.GetHealthSnapshot(); + h.TotalQueries.ShouldBe(2); + h.TotalSuccesses.ShouldBe(2); + h.ConsecutiveFailures.ShouldBe(0); + h.LastSuccessTime.ShouldNotBeNull(); + h.ProcessConnectionOpen.ShouldBeTrue(); + } +} diff --git a/tests/Drivers/Historian.Conformance/TagNameConformance.cs b/tests/Drivers/Historian.Conformance/TagNameConformance.cs new file mode 100644 index 00000000..46054377 --- /dev/null +++ b/tests/Drivers/Historian.Conformance/TagNameConformance.cs @@ -0,0 +1,64 @@ +using Shouldly; +using Xunit; +using ZB.MOM.WW.OtOpcUa.Core.Abstractions; +using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common; + +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Conformance; + +/// +/// Names longer than the 400-character tag-name limit are refused, never truncated. Each test seeds a series under +/// the 400-character PREFIX of the long name: a sized parameter would silently cut the long name to that prefix +/// and hit the wrong series. +/// +public abstract class TagNameConformance : ConformanceBase +{ + private static readonly CancellationToken Ct = CancellationToken.None; + + private const uint BadInvalidArgument = 0x80AB0000u; + private const uint BadNoData = 0x809B0000u; + + private static readonly string Prefix = new('p', 400); + private static readonly string Long = Prefix + "-x"; + + [Fact] + public async Task The_writer_refuses_a_long_name_as_a_per_tag_failure_and_writes_nothing() + { + await B.SeedAsync(Prefix, (At(0), HistorianValue.FromDouble(1), Good)); + var w = B.CreateValueWriter(); + + (await w.WriteLiveValuesAsync(Long, [new(At(5), HistorianValue.FromDouble(2), 0u)], Ct)).ShouldBeFalse(); + + (await B.ReadBackAsync(Prefix)).Count.ShouldBe(1); // nothing landed on the prefix series + (await B.TagExistsAsync(Long)).ShouldBeFalse(); // and the long tag was not created + } + + [Fact] + public async Task The_updater_answers_BadInvalidArgument_for_every_item_of_a_long_name() + { + await B.SeedAsync(Prefix, (At(0), HistorianValue.FromDouble(1), Good)); + var u = B.CreateUpdater(); + + (await u.UpdateDataAsync(Long, HistoryUpdateMode.Update, + [new(At(0), HistorianValue.FromDouble(9), 0u), new(At(1), HistorianValue.FromDouble(9), 0u)], Ct)) + .ShouldBe([BadInvalidArgument, BadInvalidArgument]); + (await u.DeleteRawAsync(Long, At(-1), At(10), Ct)).ShouldBe(BadInvalidArgument); + (await u.DeleteAtTimeAsync(Long, [At(0)], Ct)).ShouldBe([BadInvalidArgument]); + + // The prefix series is untouched: not replaced, not deleted. + var raw = (await B.CreateDataSource().ReadRawAsync(Prefix, At(-1), At(10), 0, Ct)).Samples; + raw.Select(s => s.Value).ShouldBe([(object)1d]); + } + + [Fact] + public async Task Reads_of_a_long_name_are_not_found_rather_than_the_prefix_series() + { + await B.SeedAsync(Prefix, (At(0), HistorianValue.FromDouble(1), Good)); + var src = B.CreateDataSource(); + + (await src.ReadRawAsync(Long, At(-1), At(10), 0, Ct)).Samples.ShouldBeEmpty(); + (await src.ReadAtTimeAsync(Long, [At(0)], Ct)).Samples.Single().StatusCode.ShouldBe(BadNoData); + (await src.ReadProcessedAsync(Long, At(-1), At(10), TimeSpan.Zero, HistoryAggregateType.Count, Ct)) + .Samples.Single().StatusCode.ShouldBe(HistorianInfoBits.WithCalculated(BadNoData)); + src.GetHealthSnapshot().TotalFailures.ShouldBe(0); + } +} diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/UpdateTests.cs b/tests/Drivers/Historian.Conformance/UpdateConformance.cs similarity index 62% rename from tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/UpdateTests.cs rename to tests/Drivers/Historian.Conformance/UpdateConformance.cs index 98001164..4d60029e 100644 --- a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/UpdateTests.cs +++ b/tests/Drivers/Historian.Conformance/UpdateConformance.cs @@ -1,205 +1,173 @@ -using Microsoft.Data.SqlClient; using Shouldly; using Xunit; using ZB.MOM.WW.OtOpcUa.Core.Abstractions; using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common; -namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests; +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Conformance; -[Collection(SqlHistorianCollection.Name)] -public sealed class UpdateTests(SqlHistorianFixture fx) +/// History update/delete result codes and their effect on storage. +public abstract class UpdateConformance : ConformanceBase { - private const uint Good = HistorianStatusCodes.Good; private const uint UncertainStatus = HistorianStatusCodes.UncertainDataSubNormal; - // Deliberately NOT millisecond aligned: sub-ms ticks must survive the TVP, MERGE join and DELETE join. + // Deliberately NOT millisecond (or microsecond) aligned: sub-resolution ticks must survive; stored times are B.Round(T(n)). private static readonly DateTime T0 = new(2026, 1, 1, 0, 0, 0, DateTimeKind.Utc); private static DateTime T(int n) => T0.AddTicks(1234567L * n + 3); - private (SqlServerHistorianDataUpdater U, string Schema) Make() - { - fx.SkipIfUnavailable(); - var schema = SqlHistorianFixture.UniqueSchema(); - var o = new SqlServerHistorianOptions { ConnectionString = fx.ConnectionString, Schema = schema }; - var conn = new SqlHistorianConnection(o); - var boot = new SqlServerSchemaBootstrapper(conn, o); - return (new SqlServerHistorianDataUpdater(conn, new SqlServerTagCatalog(conn, boot)), schema); - } - private static HistorizationValue V(DateTime? t, double d, uint status = Good) => new(t, HistorianValue.FromDouble(d), status); - private async Task> RowsAsync(string schema, string tag) - { - await using var c = new SqlConnection(fx.ConnectionString); - await c.OpenAsync(); - await using var cmd = c.CreateCommand(); - cmd.CommandText = $"SELECT s.SourceTimeUtc, s.NumValue, s.StatusCode FROM [{schema}].Sample s JOIN [{schema}].Tag g ON g.TagId=s.TagId WHERE g.Name=@n ORDER BY s.SourceTimeUtc"; - cmd.Parameters.AddWithValue("@n", tag); - var list = new List<(DateTime, double?, int)>(); - await using var rd = await cmd.ExecuteReaderAsync(); - while (await rd.ReadAsync()) - list.Add((DateTime.SpecifyKind(rd.GetDateTime(0), DateTimeKind.Utc), rd.IsDBNull(1) ? null : rd.GetDouble(1), rd.GetInt32(2))); - return list; - } - - private async Task TagCountAsync(string schema) - { - await using var c = new SqlConnection(fx.ConnectionString); - await c.OpenAsync(); - await using var cmd = c.CreateCommand(); - cmd.CommandText = $"SELECT COUNT(*) FROM [{schema}].Tag"; - return Convert.ToInt64(await cmd.ExecuteScalarAsync()); - } - private static readonly CancellationToken Ct = CancellationToken.None; [Fact] public async Task Insert_new_is_GoodEntryInserted_and_existing_is_BadEntryExists_in_order() { - var (u, s) = Make(); + var u = B.CreateUpdater(); var r1 = await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T(1), 1), V(T(2), 2)], Ct); r1.ShouldBe([HistorianStatusCodes.GoodEntryInserted, HistorianStatusCodes.GoodEntryInserted]); // Mixed: T(2) exists, T(3) new, T(1) exists - out of timestamp order on purpose. var r2 = await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T(3), 3), V(T(2), 99), V(T(1), 98)], Ct); r2.ShouldBe([HistorianStatusCodes.GoodEntryInserted, HistorianStatusCodes.BadEntryExists, HistorianStatusCodes.BadEntryExists]); - var rows = await RowsAsync(s, "t"); - rows.Select(x => x.Num).ShouldBe([1d, 2d, 3d]); // existing values untouched - rows.Select(x => x.Time).ShouldBe([T(1), T(2), T(3)]); // sub-ms ticks preserved + var rows = await B.ReadBackAsync("t"); + rows.Select(x => (double?)x.V.Number).ShouldBe([1d, 2d, 3d]); // existing values untouched + rows.Select(x => x.T).ShouldBe([B.Round(T(1)), B.Round(T(2)), B.Round(T(3))]); // sub-ms ticks preserved } [Fact] public async Task Replace_existing_is_GoodEntryReplaced_with_new_value_and_status_missing_is_BadNoEntryExists() { - var (u, s) = Make(); + var u = B.CreateUpdater(); await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T(1), 1), V(T(2), 2)], Ct); var r = await u.UpdateDataAsync("t", HistoryUpdateMode.Replace, [V(T(2), 22, UncertainStatus), V(T(5), 5), V(T(1), 11)], Ct); r.ShouldBe([HistorianStatusCodes.GoodEntryReplaced, HistorianStatusCodes.BadNoEntryExists, HistorianStatusCodes.GoodEntryReplaced]); - var rows = await RowsAsync(s, "t"); + var rows = await B.ReadBackAsync("t"); rows.Count.ShouldBe(2); // T(5) was not created - rows[0].Num.ShouldBe(11d); - rows[0].Status.ShouldBe((int)Good); - rows[1].Num.ShouldBe(22d); - rows[1].Status.ShouldBe(unchecked((int)UncertainStatus)); // replaced value visible with its new status + rows[0].V.Number.ShouldBe(11d); + rows[0].Status.ShouldBe(Good); + rows[1].V.Number.ShouldBe(22d); + rows[1].Status.ShouldBe(UncertainStatus); // replaced value visible with its new status } [Fact] public async Task Update_upserts_inserted_or_replaced_per_item() { - var (u, s) = Make(); + var u = B.CreateUpdater(); await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T(1), 1)], Ct); var r = await u.UpdateDataAsync("t", HistoryUpdateMode.Update, [V(T(2), 2), V(T(1), 10)], Ct); r.ShouldBe([HistorianStatusCodes.GoodEntryInserted, HistorianStatusCodes.GoodEntryReplaced]); - (await RowsAsync(s, "t")).Select(x => x.Num).ShouldBe([10d, 2d]); + (await B.ReadBackAsync("t")).Select(x => (double?)x.V.Number).ShouldBe([10d, 2d]); } [Fact] public async Task Unknown_tag_Insert_and_Update_create_it_but_Replace_and_deletes_do_not() { - var (u, s) = Make(); + var u = B.CreateUpdater(); (await u.UpdateDataAsync("rep", HistoryUpdateMode.Replace, [V(T(1), 1)], Ct)) .ShouldBe([HistorianStatusCodes.BadNoEntryExists]); (await u.DeleteRawAsync("del", T(0), T(9), Ct)).ShouldBe(HistorianStatusCodes.BadNoData); (await u.DeleteAtTimeAsync("del2", [T(1)], Ct)).ShouldBe([HistorianStatusCodes.BadNoEntryExists]); - (await TagCountAsync(s)).ShouldBe(0); // none of those created a tag + foreach (var name in new[] { "rep", "del", "del2" }) + (await B.TagExistsAsync(name)).ShouldBeFalse(name); // none of those created a tag (await u.UpdateDataAsync("ins", HistoryUpdateMode.Insert, [V(T(1), 1)], Ct)).ShouldBe([HistorianStatusCodes.GoodEntryInserted]); (await u.UpdateDataAsync("upd", HistoryUpdateMode.Update, [V(T(1), 1)], Ct)).ShouldBe([HistorianStatusCodes.GoodEntryInserted]); - (await TagCountAsync(s)).ShouldBe(2); + (await B.TagExistsAsync("ins")).ShouldBeTrue(); + (await B.TagExistsAsync("upd")).ShouldBeTrue(); } [Fact] public async Task Null_and_MinValue_timestamps_are_BadInvalidTimestamp_and_others_proceed() { - var (u, s) = Make(); + var u = B.CreateUpdater(); var r = await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(null, 1), V(T(1), 2), V(DateTime.MinValue, 3)], Ct); r.ShouldBe([HistorianStatusCodes.BadInvalidTimestamp, HistorianStatusCodes.GoodEntryInserted, HistorianStatusCodes.BadInvalidTimestamp]); - (await RowsAsync(s, "t")).Count.ShouldBe(1); + (await B.ReadBackAsync("t")).Count.ShouldBe(1); // All-invalid request: no DB work, no tag created, still one code per item. - var (u2, s2) = Make(); + var u2 = B.CreateUpdater(); await u2.UpdateDataAsync("seed", HistoryUpdateMode.Insert, [V(T(1), 1)], Ct); // bootstraps the schema (await u2.UpdateDataAsync("x", HistoryUpdateMode.Insert, [V(null, 1)], Ct)).ShouldBe([HistorianStatusCodes.BadInvalidTimestamp]); - (await TagCountAsync(s2)).ShouldBe(1); // only "seed"; "x" was never created + (await B.TagExistsAsync("seed")).ShouldBeTrue(); + (await B.TagExistsAsync("x")).ShouldBeFalse(); // "x" was never created } [Fact] public async Task NaN_and_Infinity_are_rejected_with_BadInvalidArgument_and_others_proceed() { - var (u, s) = Make(); + var u = B.CreateUpdater(); var r = await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T(1), double.NaN), V(T(2), 2), V(T(3), double.PositiveInfinity), V(T(4), double.NegativeInfinity)], Ct); r.ShouldBe([HistorianStatusCodes.BadInvalidArgument, HistorianStatusCodes.GoodEntryInserted, HistorianStatusCodes.BadInvalidArgument, HistorianStatusCodes.BadInvalidArgument]); - (await RowsAsync(s, "t")).Select(x => x.Num).ShouldBe([2d]); + (await B.ReadBackAsync("t")).Select(x => (double?)x.V.Number).ShouldBe([2d]); } [Fact] public async Task Duplicate_timestamp_in_one_Insert_request_second_is_BadEntryExists_first_wins() { - var (u, s) = Make(); + var u = B.CreateUpdater(); var r = await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T(1), 1), V(T(2), 2), V(T(1), 99)], Ct); r.ShouldBe([HistorianStatusCodes.GoodEntryInserted, HistorianStatusCodes.GoodEntryInserted, HistorianStatusCodes.BadEntryExists]); - (await RowsAsync(s, "t")).Select(x => x.Num).ShouldBe([1d, 2d]); + (await B.ReadBackAsync("t")).Select(x => (double?)x.V.Number).ShouldBe([1d, 2d]); } [Fact] public async Task Duplicate_timestamp_in_one_Replace_or_Update_request_last_wins_and_all_share_the_code() { - var (u, s) = Make(); + var u = B.CreateUpdater(); await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T(1), 1)], Ct); var rep = await u.UpdateDataAsync("t", HistoryUpdateMode.Replace, [V(T(1), 5), V(T(1), 6)], Ct); rep.ShouldBe([HistorianStatusCodes.GoodEntryReplaced, HistorianStatusCodes.GoodEntryReplaced]); - (await RowsAsync(s, "t")).Single().Num.ShouldBe(6d); + (await B.ReadBackAsync("t")).Single().V.Number.ShouldBe(6d); var upd = await u.UpdateDataAsync("t", HistoryUpdateMode.Update, [V(T(7), 7), V(T(7), 8)], Ct); upd.ShouldBe([HistorianStatusCodes.GoodEntryInserted, HistorianStatusCodes.GoodEntryInserted]); - (await RowsAsync(s, "t")).Last().Num.ShouldBe(8d); + (await B.ReadBackAsync("t")).Last().V.Number.ShouldBe(8d); } [Fact] public async Task Replace_and_Update_can_change_the_value_kind_and_clear_stale_columns() { - var (u, s) = Make(); + var u = B.CreateUpdater(); await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [new HistorizationValue(T(1), HistorianValue.FromDouble(1.5), Good)], Ct); (await u.UpdateDataAsync("t", HistoryUpdateMode.Replace, [new HistorizationValue(T(1), HistorianValue.FromString("hi"), Good)], Ct)).ShouldBe([HistorianStatusCodes.GoodEntryReplaced]); - var rows = await RowsAsync(s, "t"); - rows.Single().Num.ShouldBeNull(); // NumValue cleared by the String replace + var rows = await B.ReadBackAsync("t"); + rows.Single().V.ShouldBe(HistorianValue.FromString("hi")); // Number cleared (0), not left at 1.5 } [Fact] public async Task DeleteRaw_is_half_open_Good_when_rows_removed_BadNoData_when_none() { - var (u, s) = Make(); + var u = B.CreateUpdater(); await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T(1), 1), V(T(2), 2), V(T(3), 3)], Ct); // [T(1), T(3)) removes T(1) and T(2); T(3) (the end) survives. (await u.DeleteRawAsync("t", T(1), T(3), Ct)).ShouldBe(Good); - (await RowsAsync(s, "t")).Select(x => x.Time).ShouldBe([T(3)]); + (await B.ReadBackAsync("t")).Select(x => x.T).ShouldBe([B.Round(T(3))]); (await u.DeleteRawAsync("t", T(1), T(3), Ct)).ShouldBe(HistorianStatusCodes.BadNoData); - // sub-ms boundary: a start one tick after T(3) excludes it. - (await u.DeleteRawAsync("t", T(3).AddTicks(1), T(3).AddTicks(100), Ct)).ShouldBe(HistorianStatusCodes.BadNoData); - (await RowsAsync(s, "t")).Count.ShouldBe(1); + // sub-ms boundary: a start one resolution unit after T(3) excludes it. + (await u.DeleteRawAsync("t", T(3).AddTicks(B.ResolutionTicks), T(3).AddTicks(100 * B.ResolutionTicks), Ct)).ShouldBe(HistorianStatusCodes.BadNoData); + (await B.ReadBackAsync("t")).Count.ShouldBe(1); } [Fact] public async Task DeleteAtTime_reports_per_time_in_order_with_exact_tick_matching() { - var (u, s) = Make(); + var u = B.CreateUpdater(); await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T(1), 1), V(T(2), 2), V(T(3), 3)], Ct); - var r = await u.DeleteAtTimeAsync("t", [T(3), T(2).AddTicks(1) /* near miss */, T(1), DateTime.MinValue], Ct); + var r = await u.DeleteAtTimeAsync("t", [T(3), T(2).AddTicks(B.ResolutionTicks) /* near miss */, T(1), DateTime.MinValue], Ct); r.ShouldBe([Good, HistorianStatusCodes.BadNoEntryExists, Good, HistorianStatusCodes.BadInvalidTimestamp]); - (await RowsAsync(s, "t")).Select(x => x.Time).ShouldBe([T(2)]); + (await B.ReadBackAsync("t")).Select(x => x.T).ShouldBe([B.Round(T(2))]); } [Fact] public async Task DeleteAtTime_duplicate_time_first_reports_Good_later_BadNoEntryExists() { - var (u, s) = Make(); + var u = B.CreateUpdater(); await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T(1), 1)], Ct); (await u.DeleteAtTimeAsync("t", [T(1), T(1)], Ct)).ShouldBe([Good, HistorianStatusCodes.BadNoEntryExists]); - (await RowsAsync(s, "t")).ShouldBeEmpty(); + (await B.ReadBackAsync("t")).ShouldBeEmpty(); } } diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/LongNameTests.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/LongNameTests.cs index 1b34e04d..665f9dfa 100644 --- a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/LongNameTests.cs +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/LongNameTests.cs @@ -16,57 +16,9 @@ public sealed class LongNameTests(SqlHistorianFixture fx) private static readonly CancellationToken Ct = CancellationToken.None; private static DateTime At(double s) => ReadTestDb.At(s); - private const uint BadInvalidArgument = 0x80AB0000u; - private const uint BadNoData = 0x809B0000u; - private static readonly string Prefix = new('p', 400); private static readonly string Long = Prefix + "-x"; - [Fact] - public async Task The_writer_refuses_a_long_name_as_a_per_tag_failure_and_writes_nothing() - { - var db = await ReadTestDb.CreateAsync(fx); - await db.SeedAsync(Prefix, (At(0), HistorianValue.FromDouble(1), ReadTestDb.Good)); - var w = new SqlServerHistorianValueWriter(db.Conn, db.Boot, new SqlServerTagCatalog(db.Conn, db.Boot)); - - (await w.WriteLiveValuesAsync(Long, [new(At(5), HistorianValue.FromDouble(2), 0u)], Ct)).ShouldBeFalse(); - - (await db.ScalarAsync($"SELECT COUNT(*) FROM {db.Conn.Q("Sample")}")).ShouldBe(1); - (await db.ScalarAsync($"SELECT COUNT(*) FROM {db.Conn.Q("Tag")}")).ShouldBe(1); - } - - [Fact] - public async Task The_updater_answers_BadInvalidArgument_for_every_item_of_a_long_name() - { - var db = await ReadTestDb.CreateAsync(fx); - await db.SeedAsync(Prefix, (At(0), HistorianValue.FromDouble(1), ReadTestDb.Good)); - var u = new SqlServerHistorianDataUpdater(db.Conn, new SqlServerTagCatalog(db.Conn, db.Boot)); - - (await u.UpdateDataAsync(Long, HistoryUpdateMode.Update, - [new(At(0), HistorianValue.FromDouble(9), 0u), new(At(1), HistorianValue.FromDouble(9), 0u)], Ct)) - .ShouldBe([BadInvalidArgument, BadInvalidArgument]); - (await u.DeleteRawAsync(Long, At(-1), At(10), Ct)).ShouldBe(BadInvalidArgument); - (await u.DeleteAtTimeAsync(Long, [At(0)], Ct)).ShouldBe([BadInvalidArgument]); - - // The prefix series is untouched: not replaced, not deleted. - var raw = (await db.Source().ReadRawAsync(Prefix, At(-1), At(10), 0, Ct)).Samples; - raw.Select(s => s.Value).ShouldBe([(object)1d]); - } - - [Fact] - public async Task Reads_of_a_long_name_are_not_found_rather_than_the_prefix_series() - { - var db = await ReadTestDb.CreateAsync(fx); - await db.SeedAsync(Prefix, (At(0), HistorianValue.FromDouble(1), ReadTestDb.Good)); - var src = db.Source(); - - (await src.ReadRawAsync(Long, At(-1), At(10), 0, Ct)).Samples.ShouldBeEmpty(); - (await src.ReadAtTimeAsync(Long, [At(0)], Ct)).Samples.Single().StatusCode.ShouldBe(BadNoData); - (await src.ReadProcessedAsync(Long, At(-1), At(10), TimeSpan.Zero, HistoryAggregateType.Count, Ct)) - .Samples.Single().StatusCode.ShouldBe(HistorianInfoBits.WithCalculated(BadNoData)); - src.GetHealthSnapshot().TotalFailures.ShouldBe(0); - } - [Fact] public async Task An_event_read_filtered_by_a_long_source_matches_nothing() { diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/RawReadTests.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/RawReadTests.cs index 8d62d081..6a7a213e 100644 --- a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/RawReadTests.cs +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/RawReadTests.cs @@ -6,230 +6,12 @@ using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common; namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests; -/// -/// Seeds a fresh schema with plain INSERTs (independent of the value writer) and builds a data source over it. -/// Shared by the four read test classes. is deliberately NOT millisecond-aligned. -/// -internal sealed class ReadTestDb -{ - /// Base time: 2026-10-09 08:00:00.1234567 UTC (sub-millisecond ticks on purpose). - public static readonly DateTime T0 = new DateTime(2026, 10, 9, 8, 0, 0, DateTimeKind.Utc).AddTicks(1_234_567); - - public const uint Good = 0x00000000u; - public const uint Bad = 0x80000000u; - public const uint BadNoCommunication = 0x80310000u; - public const uint Uncertain = 0x40000000u; - - private ReadTestDb(SqlServerHistorianOptions options, SqlHistorianConnection conn, SqlServerSchemaBootstrapper boot) - { - Options = options; - Conn = conn; - Boot = boot; - } - - public SqlServerHistorianOptions Options { get; } - public SqlHistorianConnection Conn { get; } - public SqlServerSchemaBootstrapper Boot { get; } - - public static DateTime At(double seconds) => T0.AddTicks((long)(seconds * TimeSpan.TicksPerSecond)); - - public static async Task CreateAsync(SqlHistorianFixture fx) - { - fx.SkipIfUnavailable(); - var o = new SqlServerHistorianOptions { ConnectionString = fx.ConnectionString, Schema = SqlHistorianFixture.UniqueSchema() }; - var conn = new SqlHistorianConnection(o); - var boot = new SqlServerSchemaBootstrapper(conn, o); - await boot.EnsureAsync(CancellationToken.None); - return new ReadTestDb(o, conn, boot); - } - - /// A fresh data source (fresh catalog cache) over this schema; is its "now". - public SqlServerHistorianDataSource Source(TimeProvider? clock = null) => - new(Conn, Boot, new SqlServerTagCatalog(Conn, Boot), timeProvider: clock); - - /// Inserts a tag row and the given samples. ServerTimeUtc = source time + 1 s (so tests can tell them apart). - public async Task SeedAsync(string tag, params (DateTime T, HistorianValue V, uint Status)[] samples) - { - await using var c = await Conn.OpenAsync(CancellationToken.None); - int id; - await using (var cmd = Conn.CreateCommand(c, - $"INSERT {Conn.Q("Tag")} (Name, DataType, CreatedUtc, UpdatedUtc) OUTPUT inserted.TagId VALUES (@n, 0, SYSUTCDATETIME(), SYSUTCDATETIME())")) - { - cmd.Parameters.AddWithValue("@n", tag); - id = (int)(await cmd.ExecuteScalarAsync())!; - } - foreach (var (t, v, status) in samples) - { - var col = SampleColumnMapper.ToColumns(v); - await using var cmd = Conn.CreateCommand(c, - $"INSERT {Conn.Q("Sample")} (TagId, SourceTimeUtc, ValueKind, NumValue, IntValue, StrValue, StatusCode, ServerTimeUtc) " + - "VALUES (@id, @t, @k, @num, @int, @str, @st, @srv)"); - cmd.Parameters.AddWithValue("@id", id); - SqlHistorianConnection.AddDateTime2(cmd, "@t", t); - cmd.Parameters.AddWithValue("@k", col.ValueKind); - cmd.Parameters.Add(new SqlParameter("@num", System.Data.SqlDbType.Float) { Value = (object?)col.NumValue ?? DBNull.Value }); - cmd.Parameters.Add(new SqlParameter("@int", System.Data.SqlDbType.BigInt) { Value = (object?)col.IntValue ?? DBNull.Value }); - cmd.Parameters.Add(new SqlParameter("@str", System.Data.SqlDbType.NVarChar, -1) { Value = (object?)col.StrValue ?? DBNull.Value }); - cmd.Parameters.AddWithValue("@st", SampleColumnMapper.ToDbStatus(status)); - SqlHistorianConnection.AddDateTime2(cmd, "@srv", t.AddSeconds(1)); - await cmd.ExecuteNonQueryAsync(); - } - return id; - } - - /// The fixture tag A of the brief: +0=10, +10=20, +20=Bad(30), +30=40, +40=50 (Double). - public Task SeedAAsync(string name = "A") => SeedAsync(name, - (At(0), HistorianValue.FromDouble(10), Good), - (At(10), HistorianValue.FromDouble(20), Good), - (At(20), HistorianValue.FromDouble(30), Bad), - (At(30), HistorianValue.FromDouble(40), Good), - (At(40), HistorianValue.FromDouble(50), Good)); - - public async Task ScalarAsync(string sql) - { - await using var c = await Conn.OpenAsync(CancellationToken.None); - await using var cmd = Conn.CreateCommand(c, sql); - return Convert.ToInt64(await cmd.ExecuteScalarAsync()); - } -} - [Collection(SqlHistorianCollection.Name)] public sealed class RawReadTests(SqlHistorianFixture fx) { private static readonly CancellationToken Ct = CancellationToken.None; private static DateTime At(double s) => ReadTestDb.At(s); - private static double?[] Values(HistoryReadResult r) => r.Samples.Select(s => (double?)s.Value).ToArray(); - - [Fact] - public async Task Forward_read_is_start_inclusive_end_exclusive_ascending_with_exact_ticks() - { - var db = await ReadTestDb.CreateAsync(fx); - await db.SeedAAsync(); - var r = await db.Source().ReadRawAsync("A", At(10), At(40), 0, Ct); - - Values(r).ShouldBe([20d, 30d, 40d]); // +10 included, +40 (== end) excluded - r.Samples.Select(s => s.SourceTimestampUtc!.Value.Ticks).ShouldBe([At(10).Ticks, At(20).Ticks, At(30).Ticks]); - r.Samples.Select(s => s.StatusCode).ShouldBe([ReadTestDb.Good, ReadTestDb.Bad, ReadTestDb.Good]); - r.Samples[0].SourceTimestampUtc!.Value.Kind.ShouldBe(DateTimeKind.Utc); - r.Samples[0].ServerTimestampUtc.Ticks.ShouldBe(At(11).Ticks); // stored ServerTimeUtc, not "now" - r.ContinuationPoint.ShouldBeNull(); - } - - [Fact] - public async Task Forward_read_honours_the_limit() - { - var db = await ReadTestDb.CreateAsync(fx); - await db.SeedAAsync(); - Values(await db.Source().ReadRawAsync("A", At(0), At(100), 2, Ct)).ShouldBe([10d, 20d]); - } - - [Fact] - public async Task Zero_max_values_is_unbounded() - { - var db = await ReadTestDb.CreateAsync(fx); - await db.SeedAAsync(); - Values(await db.Source().ReadRawAsync("A", At(-1), At(100), 0, Ct)).ShouldBe([10d, 20d, 30d, 40d, 50d]); - Values(await db.Source().ReadRawAsync("A", At(-1), At(100), uint.MaxValue, Ct)).Length.ShouldBe(5); - } - - [Fact] - public async Task Reverse_read_is_descending_over_end_exclusive_start_inclusive() - { - var db = await ReadTestDb.CreateAsync(fx); - await db.SeedAAsync(); - // start=+30, end=+10 ⇒ (+10, +30] descending. - Values(await db.Source().ReadRawAsync("A", At(30), At(10), 0, Ct)).ShouldBe([40d, 30d]); - Values(await db.Source().ReadRawAsync("A", At(100), At(-1), 2, Ct)).ShouldBe([50d, 40d]); - } - - [Fact] - public async Task Equal_start_and_end_is_a_point_read() - { - var db = await ReadTestDb.CreateAsync(fx); - await db.SeedAAsync(); - Values(await db.Source().ReadRawAsync("A", At(10), At(10), 0, Ct)).ShouldBe([20d]); - (await db.Source().ReadRawAsync("A", At(10).AddTicks(1), At(10).AddTicks(1), 0, Ct)).Samples.ShouldBeEmpty(); - } - - [Fact] - public async Task A_resumed_page_from_the_last_timestamp_inclusive_continues_without_gaps() - { - // The node manager's paging: read cap N, resume at the last returned timestamp INCLUSIVE, trim it. - var db = await ReadTestDb.CreateAsync(fx); - await db.SeedAAsync(); - var src = db.Source(); - var p1 = await src.ReadRawAsync("A", At(0), At(100), 2, Ct); - var p2 = await src.ReadRawAsync("A", p1.Samples[^1].SourceTimestampUtc!.Value, At(100), 2, Ct); - Values(p1).ShouldBe([10d, 20d]); - Values(p2).ShouldBe([20d, 30d]); // first is the boundary sample the node manager trims - } - - [Fact] - public async Task Unknown_tag_is_an_empty_result_and_reads_never_create_tags() - { - var db = await ReadTestDb.CreateAsync(fx); - var src = db.Source(); - (await src.ReadRawAsync("nope", At(0), At(10), 0, Ct)).Samples.ShouldBeEmpty(); - await src.ReadAtTimeAsync("nope", [At(1)], Ct); - await src.ReadProcessedAsync("nope", At(0), At(10), TimeSpan.FromSeconds(5), HistoryAggregateType.Average, Ct); - (await db.ScalarAsync($"SELECT COUNT(*) FROM {db.Conn.Q("Tag")}")).ShouldBe(0); - src.GetHealthSnapshot().TotalFailures.ShouldBe(0); - } - - [Fact] - public async Task Outage_markers_and_bad_samples_come_back_with_their_bad_status() - { - var db = await ReadTestDb.CreateAsync(fx); - await db.SeedAsync("M", - (At(0), HistorianValue.FromDouble(10), ReadTestDb.Good), - (At(5), HistorianValue.Empty, ReadTestDb.BadNoCommunication), - (At(7), HistorianValue.FromDouble(99), ReadTestDb.Bad)); - var r = await db.Source().ReadRawAsync("M", At(0), At(10), 0, Ct); - r.Samples.Count.ShouldBe(3); - r.Samples[1].Value.ShouldBeNull(); - r.Samples[1].StatusCode.ShouldBe(0x80310000u); - r.Samples[2].Value.ShouldBe(99d); - r.Samples[2].StatusCode.ShouldBe(0x80000000u); - } - - [Fact] - public async Task Every_kind_round_trips_to_its_CLR_type() - { - var db = await ReadTestDb.CreateAsync(fx); - var dt = new DateTime(2025, 1, 2, 3, 4, 5, DateTimeKind.Utc).AddTicks(7); - await db.SeedAsync("K", - (At(0), HistorianValue.FromInt64(-9_000_000_000L), ReadTestDb.Good), - (At(1), HistorianValue.FromBoolean(true), ReadTestDb.Good), - (At(2), HistorianValue.FromString("héllo"), ReadTestDb.Good), - (At(3), HistorianValue.FromDateTime(dt), ReadTestDb.Good), - (At(4), HistorianValue.FromDouble(1.5), ReadTestDb.Uncertain)); - var s = (await db.Source().ReadRawAsync("K", At(0), At(5), 0, Ct)).Samples; - s[0].Value.ShouldBe(-9_000_000_000L); - s[1].Value.ShouldBe(true); - s[2].Value.ShouldBe("héllo"); - s[3].Value.ShouldBe(dt); - ((DateTime)s[3].Value!).Kind.ShouldBe(DateTimeKind.Utc); - s[4].Value.ShouldBe(1.5); - s[4].StatusCode.ShouldBe(0x40000000u); - } - - [Fact] - public async Task Successful_reads_are_counted_in_health() - { - var db = await ReadTestDb.CreateAsync(fx); - await db.SeedAAsync(); - var src = db.Source(); - await src.ReadRawAsync("A", At(0), At(10), 0, Ct); - await src.ReadRawAsync("A", At(0), At(10), 0, Ct); - var h = src.GetHealthSnapshot(); - h.TotalQueries.ShouldBe(2); - h.TotalSuccesses.ShouldBe(2); - h.ConsecutiveFailures.ShouldBe(0); - h.LastSuccessTime.ShouldNotBeNull(); - h.ProcessConnectionOpen.ShouldBeTrue(); - } - [Fact] public async Task A_missing_schema_surfaces_in_health_LastError_and_is_not_latched() { diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/ReadTestDb.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/ReadTestDb.cs new file mode 100644 index 00000000..8af5e084 --- /dev/null +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/ReadTestDb.cs @@ -0,0 +1,87 @@ +using Microsoft.Data.SqlClient; +using Shouldly; +using Xunit; +using ZB.MOM.WW.OtOpcUa.Core.Abstractions; +using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common; + +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests; + +/// +/// Seeds a fresh schema with plain INSERTs (independent of the value writer) and builds a data source over it. +/// Shared by the SQL Server-specific tests and by . is deliberately NOT millisecond-aligned. +/// +internal sealed class ReadTestDb +{ + /// Base time: 2026-10-09 08:00:00.1234567 UTC (sub-millisecond ticks on purpose). + public static readonly DateTime T0 = Conformance.ConformanceData.T0; + + public const uint Good = Conformance.ConformanceData.Good; + public const uint Bad = Conformance.ConformanceData.Bad; + public const uint BadNoCommunication = Conformance.ConformanceData.BadNoCommunication; + public const uint Uncertain = Conformance.ConformanceData.Uncertain; + + private ReadTestDb(SqlServerHistorianOptions options, SqlHistorianConnection conn, SqlServerSchemaBootstrapper boot) + { + Options = options; + Conn = conn; + Boot = boot; + } + + public SqlServerHistorianOptions Options { get; } + public SqlHistorianConnection Conn { get; } + public SqlServerSchemaBootstrapper Boot { get; } + + public static DateTime At(double seconds) => Conformance.ConformanceData.At(seconds); + + public static async Task CreateAsync(SqlHistorianFixture fx) + { + fx.SkipIfUnavailable(); + var o = new SqlServerHistorianOptions { ConnectionString = fx.ConnectionString, Schema = SqlHistorianFixture.UniqueSchema() }; + var conn = new SqlHistorianConnection(o); + var boot = new SqlServerSchemaBootstrapper(conn, o); + await boot.EnsureAsync(CancellationToken.None); + return new ReadTestDb(o, conn, boot); + } + + /// A fresh data source (fresh catalog cache) over this schema; is its "now". + public SqlServerHistorianDataSource Source(TimeProvider? clock = null) => + new(Conn, Boot, new SqlServerTagCatalog(Conn, Boot), timeProvider: clock); + + /// Inserts a tag row and the given samples. ServerTimeUtc = source time + 1 s (so tests can tell them apart). + public async Task SeedAsync(string tag, params (DateTime T, HistorianValue V, uint Status)[] samples) + { + await using var c = await Conn.OpenAsync(CancellationToken.None); + int id; + await using (var cmd = Conn.CreateCommand(c, + $"INSERT {Conn.Q("Tag")} (Name, DataType, CreatedUtc, UpdatedUtc) OUTPUT inserted.TagId VALUES (@n, 0, SYSUTCDATETIME(), SYSUTCDATETIME())")) + { + cmd.Parameters.AddWithValue("@n", tag); + id = (int)(await cmd.ExecuteScalarAsync())!; + } + foreach (var (t, v, status) in samples) + { + var col = SampleColumnMapper.ToColumns(v); + await using var cmd = Conn.CreateCommand(c, + $"INSERT {Conn.Q("Sample")} (TagId, SourceTimeUtc, ValueKind, NumValue, IntValue, StrValue, StatusCode, ServerTimeUtc) " + + "VALUES (@id, @t, @k, @num, @int, @str, @st, @srv)"); + cmd.Parameters.AddWithValue("@id", id); + SqlHistorianConnection.AddDateTime2(cmd, "@t", t); + cmd.Parameters.AddWithValue("@k", col.ValueKind); + cmd.Parameters.Add(new SqlParameter("@num", System.Data.SqlDbType.Float) { Value = (object?)col.NumValue ?? DBNull.Value }); + cmd.Parameters.Add(new SqlParameter("@int", System.Data.SqlDbType.BigInt) { Value = (object?)col.IntValue ?? DBNull.Value }); + cmd.Parameters.Add(new SqlParameter("@str", System.Data.SqlDbType.NVarChar, -1) { Value = (object?)col.StrValue ?? DBNull.Value }); + cmd.Parameters.AddWithValue("@st", SampleColumnMapper.ToDbStatus(status)); + SqlHistorianConnection.AddDateTime2(cmd, "@srv", t.AddSeconds(1)); + await cmd.ExecuteNonQueryAsync(); + } + return id; + } + + public async Task ScalarAsync(string sql) + { + await using var c = await Conn.OpenAsync(CancellationToken.None); + await using var cmd = Conn.CreateCommand(c, sql); + return Convert.ToInt64(await cmd.ExecuteScalarAsync()); + } +} + diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/SqlServerConformance.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/SqlServerConformance.cs new file mode 100644 index 00000000..5a02a4c5 --- /dev/null +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/SqlServerConformance.cs @@ -0,0 +1,55 @@ +using ZB.MOM.WW.OtOpcUa.Driver.Historian.Conformance; +using Xunit; + +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests; + +// One concrete subclass per conformance class: the suite's tests run here against SQL Server. + +[Collection(SqlHistorianCollection.Name)] +public sealed class SqlServerRawReadConformance(SqlHistorianFixture fx) : RawReadConformance +{ + /// + protected override Task CreateBackendAsync() => SqlServerConformanceBackend.CreateAsync(fx); +} + +[Collection(SqlHistorianCollection.Name)] +public sealed class SqlServerAtTimeConformance(SqlHistorianFixture fx) : AtTimeConformance +{ + /// + protected override Task CreateBackendAsync() => SqlServerConformanceBackend.CreateAsync(fx); +} + +[Collection(SqlHistorianCollection.Name)] +public sealed class SqlServerProcessedConformance(SqlHistorianFixture fx) : ProcessedConformance +{ + /// + protected override Task CreateBackendAsync() => SqlServerConformanceBackend.CreateAsync(fx); +} + +[Collection(SqlHistorianCollection.Name)] +public sealed class SqlServerQualityContractConformance(SqlHistorianFixture fx) : QualityContractConformance +{ + /// + protected override Task CreateBackendAsync() => SqlServerConformanceBackend.CreateAsync(fx); +} + +[Collection(SqlHistorianCollection.Name)] +public sealed class SqlServerLiveWriteConformance(SqlHistorianFixture fx) : LiveWriteConformance +{ + /// + protected override Task CreateBackendAsync() => SqlServerConformanceBackend.CreateAsync(fx); +} + +[Collection(SqlHistorianCollection.Name)] +public sealed class SqlServerUpdateConformance(SqlHistorianFixture fx) : UpdateConformance +{ + /// + protected override Task CreateBackendAsync() => SqlServerConformanceBackend.CreateAsync(fx); +} + +[Collection(SqlHistorianCollection.Name)] +public sealed class SqlServerTagNameConformance(SqlHistorianFixture fx) : TagNameConformance +{ + /// + protected override Task CreateBackendAsync() => SqlServerConformanceBackend.CreateAsync(fx); +} diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/SqlServerConformanceBackend.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/SqlServerConformanceBackend.cs new file mode 100644 index 00000000..e4c27a8c --- /dev/null +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/SqlServerConformanceBackend.cs @@ -0,0 +1,77 @@ +using Microsoft.Data.SqlClient; +using ZB.MOM.WW.OtOpcUa.Core.Abstractions; +using ZB.MOM.WW.OtOpcUa.Driver.Historian.Conformance; +using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common; + +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests; + +/// +/// The SQL Server historian as a conformance backend: a fresh unique schema per instance (inside the fixture's +/// throwaway database), seeded by plain INSERTs through . +/// +internal sealed class SqlServerConformanceBackend : IConformanceBackend +{ + private readonly ReadTestDb _db; + + private SqlServerConformanceBackend(ReadTestDb db) => _db = db; + + /// Creates a backend over a fresh schema; skips the calling test when the gate is closed. + public static async Task CreateAsync(SqlHistorianFixture fx) => + new SqlServerConformanceBackend(await ReadTestDb.CreateAsync(fx)); + + /// + public long ResolutionTicks => 1; + + /// + public IHistorianDataSource CreateDataSource(TimeProvider? clock = null) => _db.Source(clock); + + /// + public IHistorianValueWriter CreateValueWriter(Func? utcNow = null) => + new SqlServerHistorianValueWriter(_db.Conn, _db.Boot, new SqlServerTagCatalog(_db.Conn, _db.Boot), utcNow: utcNow); + + /// + public IHistorianDataUpdater CreateUpdater() => + new SqlServerHistorianDataUpdater(_db.Conn, new SqlServerTagCatalog(_db.Conn, _db.Boot)); + + /// + public async Task SeedAsync(string tag, params (DateTime T, HistorianValue V, uint Status)[] samples) => + await _db.SeedAsync(tag, samples); + + /// + public async Task> ReadBackAsync(string tag) + { + await using var c = await _db.Conn.OpenAsync(CancellationToken.None); + await using var cmd = _db.Conn.CreateCommand(c, + $"SELECT s.SourceTimeUtc, s.ValueKind, s.NumValue, s.IntValue, s.StrValue, s.StatusCode FROM {_db.Conn.Q("Sample")} s " + + $"JOIN {_db.Conn.Q("Tag")} t ON t.TagId = s.TagId WHERE t.Name = @n ORDER BY s.SourceTimeUtc"); + cmd.Parameters.Add(new SqlParameter("@n", System.Data.SqlDbType.NVarChar, -1) { Value = tag }); + var list = new List<(DateTime, HistorianValue, uint)>(); + await using var r = await cmd.ExecuteReaderAsync(); + while (await r.ReadAsync()) + { + var v = SampleColumnMapper.FromColumns(r.GetByte(1), r.IsDBNull(2) ? null : r.GetDouble(2), + r.IsDBNull(3) ? null : r.GetInt64(3), r.IsDBNull(4) ? null : r.GetString(4)); + list.Add((SqlHistorianConnection.GetUtc(r, 0), v, SampleColumnMapper.FromDbStatus(r.GetInt32(5)))); + } + return list; + } + + /// + public async Task TagExistsAsync(string tag) => await ScalarTagAsync("COUNT(*)", tag) is > 0; + + /// + public async Task TagDataTypeAsync(string tag) => + await ScalarTagAsync("MAX(CONVERT(bigint, DataType))", tag) is { } v ? (DriverDataType)(byte)v : null; // MAX over no row is NULL + + private async Task ScalarTagAsync(string expr, string tag) + { + await using var c = await _db.Conn.OpenAsync(CancellationToken.None); + await using var cmd = _db.Conn.CreateCommand(c, $"SELECT {expr} FROM {_db.Conn.Q("Tag")} WHERE Name = @n"); + cmd.Parameters.Add(new SqlParameter("@n", System.Data.SqlDbType.NVarChar, -1) { Value = tag }); + var o = await cmd.ExecuteScalarAsync(); + return o is null or DBNull ? null : Convert.ToInt64(o); + } + + /// + public ValueTask DisposeAsync() => ValueTask.CompletedTask; // the fixture drops the whole database +} diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/WriteTests.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/WriteTests.cs index 3945257e..60582f6f 100644 --- a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/WriteTests.cs +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/WriteTests.cs @@ -37,101 +37,6 @@ public sealed class WriteTests(SqlHistorianFixture fx) return l; } - [Fact] - public async Task Every_kind_round_trips_with_exact_ticks_and_status() - { - fx.SkipIfUnavailable(); - var (w, s) = Make(); - var dt = new DateTime(2026, 1, 2, 3, 4, 5, DateTimeKind.Utc).AddTicks(77); - HistorizationValue[] vals = - [ - new(T0, HistorianValue.FromDouble(1.25), 0), - new(T0.AddTicks(1), HistorianValue.FromInt64(long.MaxValue), 0x40000000u), - new(T0.AddTicks(2), HistorianValue.FromBoolean(true), 0), - new(T0.AddTicks(3), HistorianValue.FromString("héllo"), 0), - new(T0.AddTicks(4), HistorianValue.FromDateTime(dt), 0), - new(T0.AddTicks(5), HistorianValue.Empty, 0x80310000u), - ]; - (await w.WriteLiveValuesAsync("tag.a", vals, CancellationToken.None)).ShouldBeTrue(); - var rows = await RowsAsync(s, "tag.a"); - rows.Count.ShouldBe(6); - for (var i = 0; i < 6; i++) rows[i].T.Ticks.ShouldBe(vals[i].TimestampUtc!.Value.Ticks); - var back = rows.Select(r => SampleColumnMapper.FromColumns(r.Kind, r.N, r.I, r.S)).ToList(); - back.ShouldBe(vals.Select(v => v.Value).ToList()); - (rows[5].N, rows[5].I, rows[5].S).ShouldBe((null, null, null)); - rows[1].Status.ShouldBe(unchecked((int)0x40000000u)); - SampleColumnMapper.FromDbStatus(rows[5].Status).ShouldBe(0x80310000u); - } - - [Fact] - public async Task Writing_the_same_batch_twice_is_idempotent() - { - fx.SkipIfUnavailable(); - var (w, s) = Make(); - HistorizationValue[] vals = [new(T0, HistorianValue.FromDouble(1), 0), new(T0.AddSeconds(1), HistorianValue.FromDouble(2), 0)]; - (await w.WriteLiveValuesAsync("tag.b", vals, CancellationToken.None)).ShouldBeTrue(); - (await w.WriteLiveValuesAsync("tag.b", vals, CancellationToken.None)).ShouldBeTrue(); - // Overlapping batch: only the new row lands, the stored first value is untouched. - (await w.WriteLiveValuesAsync("tag.b", [new(T0, HistorianValue.FromDouble(99), 0), new(T0.AddSeconds(2), HistorianValue.FromDouble(3), 0)], CancellationToken.None)).ShouldBeTrue(); - var rows = await RowsAsync(s, "tag.b"); - rows.Count.ShouldBe(3); - rows[0].N.ShouldBe(1); - } - - [Fact] - public async Task Within_batch_duplicate_timestamps_do_not_fail_and_first_wins() - { - fx.SkipIfUnavailable(); - var (w, s) = Make(); - (await w.WriteLiveValuesAsync("tag.c", [new(T0, HistorianValue.FromDouble(1), 0), new(T0, HistorianValue.FromDouble(2), 0)], CancellationToken.None)).ShouldBeTrue(); - var rows = await RowsAsync(s, "tag.c"); - rows.Count.ShouldBe(1); - rows[0].N.ShouldBe(1); - } - - [Fact] - public async Task Non_finite_doubles_store_as_Empty_forced_Bad_and_do_not_fail_the_write() - { - fx.SkipIfUnavailable(); - var (w, s) = Make(); - (await w.WriteLiveValuesAsync("tag.d", [ - new(T0, HistorianValue.FromDouble(double.NaN), 0x40000000u), - new(T0.AddTicks(1), HistorianValue.FromDouble(double.PositiveInfinity), 0), - new(T0.AddTicks(2), HistorianValue.FromDouble(7), 0)], CancellationToken.None)).ShouldBeTrue(); - var rows = await RowsAsync(s, "tag.d"); - rows.Count.ShouldBe(3); - rows[0].Kind.ShouldBe((byte)HistorianValueKind.Empty); - rows[0].N.ShouldBeNull(); - rows[0].Status.ShouldBe(unchecked((int)HistorianStatusCodes.BadOutOfRange)); // Uncertain NaN ⇒ Bad (I-1) - rows[1].Status.ShouldBe(unchecked((int)HistorianStatusCodes.BadOutOfRange)); // Good +Inf ⇒ Bad (I-1) - rows[2].N.ShouldBe(7); - } - - [Fact] - public async Task Empty_batch_succeeds_without_a_call_and_null_timestamp_is_stamped() - { - fx.SkipIfUnavailable(); - var (w, s) = Make(); - (await w.WriteLiveValuesAsync("tag.e", [], CancellationToken.None)).ShouldBeTrue(); - (await w.WriteLiveValuesAsync("tag.e", [new(null, HistorianValue.FromInt64(5), 0)], CancellationToken.None)).ShouldBeTrue(); - var rows = await RowsAsync(s, "tag.e"); - rows.Count.ShouldBe(1); - (DateTime.UtcNow - rows[0].T).ShouldBeLessThan(TimeSpan.FromMinutes(1)); - } - - [Fact] - public async Task Tag_is_created_with_the_first_typed_values_data_type() - { - fx.SkipIfUnavailable(); - var (w, s) = Make(); - await w.WriteLiveValuesAsync("tag.f", [new(T0, HistorianValue.Empty, 0x80310000u), new(T0.AddTicks(1), HistorianValue.FromString("x"), 0)], CancellationToken.None); - await using var c = new SqlConnection(fx.ConnectionString); - await c.OpenAsync(); - await using var cmd = c.CreateCommand(); - cmd.CommandText = $"SELECT DataType FROM [{s}].Tag WHERE Name='tag.f'"; - Convert.ToByte(await cmd.ExecuteScalarAsync()).ShouldBe((byte)DriverDataType.String); - } - [Fact] public async Task Ten_thousand_values_write_in_under_five_seconds() { diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests.csproj b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests.csproj index 74dbd900..4da917dd 100644 --- a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests.csproj +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests.csproj @@ -21,6 +21,12 @@ + + + + + + From 11b703708cff0d1d16c9edaa8cea5a5133c1a6a4 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Sun, 11 Oct 2026 01:52:57 -0400 Subject: [PATCH 10/23] =?UTF-8?q?feat(historian):=20QuestDB=20backend=20sk?= =?UTF-8?q?eleton=20=E2=80=94=20options,=20connection,=20schema=20bootstra?= =?UTF-8?q?p=20with=20TTL?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe --- ZB.MOM.WW.OtOpcUa.slnx | 3 + .../QuestDbConnection.cs | 209 +++++++++++++++ .../QuestDbHistorianOptions.cs | 57 +++++ .../QuestDbSchemaBootstrapper.cs | 143 +++++++++++ .../QuestDbTables.cs | 83 ++++++ .../QuestDbTagCatalog.cs | 82 ++++++ .../QuestDbWalVisibility.cs | 92 +++++++ ...WW.OtOpcUa.Driver.Historian.QuestDb.csproj | 30 +++ ....WW.OtOpcUa.Core.Abstractions.Tests.csproj | 1 + .../BootstrapTests.cs | 241 ++++++++++++++++++ .../QuestDbFixture.cs | 106 ++++++++ ....Historian.QuestDb.IntegrationTests.csproj | 33 +++ .../QuestDbConnectionTests.cs | 140 ++++++++++ .../QuestDbHistorianOptionsTests.cs | 79 ++++++ .../QuestDbSchemaBootstrapperFailureTests.cs | 32 +++ .../QuestDbTablesTests.cs | 66 +++++ .../QuestDbWalVisibilityTests.cs | 33 +++ ...pcUa.Driver.Historian.QuestDb.Tests.csproj | 27 ++ 18 files changed, 1457 insertions(+) create mode 100644 src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbConnection.cs create mode 100644 src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianOptions.cs create mode 100644 src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbSchemaBootstrapper.cs create mode 100644 src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbTables.cs create mode 100644 src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbTagCatalog.cs create mode 100644 src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbWalVisibility.cs create mode 100644 src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.csproj create mode 100644 tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/BootstrapTests.cs create mode 100644 tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/QuestDbFixture.cs create mode 100644 tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests.csproj create mode 100644 tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests/QuestDbConnectionTests.cs create mode 100644 tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests/QuestDbHistorianOptionsTests.cs create mode 100644 tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests/QuestDbSchemaBootstrapperFailureTests.cs create mode 100644 tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests/QuestDbTablesTests.cs create mode 100644 tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests/QuestDbWalVisibilityTests.cs create mode 100644 tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests.csproj diff --git a/ZB.MOM.WW.OtOpcUa.slnx b/ZB.MOM.WW.OtOpcUa.slnx index 16290f6c..4621530b 100644 --- a/ZB.MOM.WW.OtOpcUa.slnx +++ b/ZB.MOM.WW.OtOpcUa.slnx @@ -27,6 +27,7 @@ + @@ -102,6 +103,8 @@ + + diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbConnection.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbConnection.cs new file mode 100644 index 00000000..1b003425 --- /dev/null +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbConnection.cs @@ -0,0 +1,209 @@ +using System.Net.Sockets; +using Npgsql; +using NpgsqlTypes; + +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb; + +/// +/// Pooled Npgsql access to QuestDB plus every T0-confirmed typing rule, in one place: how timestamps are rounded and +/// bound (R6), how BYTE columns are bound and read, and how errors are classified. +/// Contract for callers: bind every inbound through (never +/// AddWithValue), call before writing a designated timestamp, and classify +/// failures with / . Every QuestDB server error carries +/// SQLSTATE 00000, so classification is by message text, never by SQLSTATE. +/// +internal sealed class QuestDbConnection : IAsyncDisposable +{ + private static readonly long MinSafeTicks = 10; // DateTime.MinValue itself binds as -infinity + private static readonly long MaxSafeTicks = DateTime.MaxValue.Ticks - DateTime.MaxValue.Ticks % 10; // MaxValue itself binds as +infinity + + private readonly NpgsqlDataSource _dataSource; + + /// Creates the connection helper; options must already be valid. + /// Validated options. + public QuestDbConnection(QuestDbHistorianOptions options) + { + ArgumentNullException.ThrowIfNull(options); + if (!QuestDbHistorianOptions.IsValidPrefix(options.TablePrefix)) + throw new ArgumentException("Invalid table prefix.", nameof(options)); + TablePrefix = options.TablePrefix; + CommandTimeoutSeconds = Math.Max(1, (int)Math.Ceiling(options.CommandTimeout.TotalSeconds)); + _dataSource = new NpgsqlDataSourceBuilder(options.ConnectionString).Build(); + } + + /// The validated table prefix. + public string TablePrefix { get; } + + /// Command timeout applied by , in seconds. + public int CommandTimeoutSeconds { get; } + + /// Prefixes a table base name (sample, event, tag) with the configured prefix. + /// Fixed base name from this project (never user input). + /// The full table name. + public string T(string table) => TablePrefix + table; + + /// The prefixed sample table. + public string SampleTable => T(QuestDbTables.SampleBase); + /// The prefixed event table. + public string EventTable => T(QuestDbTables.EventBase); + /// The prefixed tag table. + public string TagTable => T(QuestDbTables.TagBase); + + /// Opens a pooled connection (no bootstrap; callers run the bootstrapper first). + /// Cancellation. + /// An open connection the caller disposes. + public async Task OpenAsync(CancellationToken ct) => + await _dataSource.OpenConnectionAsync(ct).ConfigureAwait(false); + + /// Creates a text command with the configured timeout. Parameters are positional ($1, $2…). + /// Open connection. + /// Command text. + /// The command. + public NpgsqlCommand CreateCommand(NpgsqlConnection conn, string sql) + { + var cmd = conn.CreateCommand(); + cmd.CommandText = sql; + cmd.CommandTimeout = CommandTimeoutSeconds; + return cmd; + } + + /// + public ValueTask DisposeAsync() => _dataSource.DisposeAsync(); + + // ---- timestamps (R6) ------------------------------------------------------------------------------------------- + + /// + /// Normalises a timestamp for QuestDB (microsecond precision). Rule: Unspecified is treated as UTC + /// (SpecifyKind); Local is converted with ToUniversalTime; then ticks are truncated to a + /// microsecond (ticks - ticks % 10, because Npgsql truncates relative to 2000-01-01 and so rounds UP + /// before 2000); finally the result is nudged into [0001-01-01T00:00:00.000001, 9999-12-31T23:59:59.999999] + /// so a value equal to MinValue/MaxValue never reaches Npgsql, which would bind it as ±infinity + /// (a [Min, Max) range then silently matches nothing). The result always has Kind=Utc. + /// + /// Any timestamp. + /// A UTC, microsecond-aligned, finite timestamp. + public static DateTime Round(DateTime value) + { + var utc = value.Kind switch + { + DateTimeKind.Local => value.ToUniversalTime(), + DateTimeKind.Utc => value, + _ => DateTime.SpecifyKind(value, DateTimeKind.Utc), + }; + var ticks = utc.Ticks - utc.Ticks % 10; + ticks = Math.Clamp(ticks, MinSafeTicks, MaxSafeTicks); + return new DateTime(ticks, DateTimeKind.Utc); + } + + /// + /// True when QuestDB can store the timestamp as a designated timestamp: after it is on or + /// after 1970-01-01 (RC1). Writers drop (live) or refuse (HistoryUpdate) samples for which this is false; reads + /// with earlier bounds are fine. + /// + /// Candidate sample time. + /// Whether an INSERT with this designated timestamp can succeed. + public static bool IsStorable(DateTime value) => Round(value) >= DateTime.UnixEpoch; + + /// + /// Appends the next positional parameter ($n) as timestamp with time zone carrying + /// 's UTC value (T0-confirmed: Timestamp+Utc throws in Npgsql). It never emits + /// MinValue/MaxValue. By default a pre-1970 value is kept as is (a pre-1970 range bound is valid + /// and clamping it would move the bound across the epoch row); pass for a + /// designated-timestamp column so a pre-1970 value can only ever become the epoch itself, never a failed batch. + /// + /// Target command. + /// Any timestamp. + /// Clamp anything before 1970-01-01 to the epoch (writes only, after ). + /// The added parameter. + public static NpgsqlParameter BindTimestamp(NpgsqlCommand cmd, DateTime value, bool clampToEpoch = false) + { + var rounded = Round(value); + if (clampToEpoch && rounded < DateTime.UnixEpoch) rounded = DateTime.UnixEpoch; + var p = new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.TimestampTz, Value = rounded }; + cmd.Parameters.Add(p); + return p; + } + + /// Appends a BYTE parameter (bound as smallint, T0-confirmed). + /// Target command. + /// The byte. + /// The added parameter. + public static NpgsqlParameter BindByte(NpgsqlCommand cmd, byte value) + { + var p = new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Smallint, Value = (short)value }; + cmd.Parameters.Add(p); + return p; + } + + /// Appends a LONG parameter (bigint; also the rule for integer arithmetic in tick math). + /// Target command. + /// The value. + /// The added parameter. + public static NpgsqlParameter BindLong(NpgsqlCommand cmd, long value) + { + var p = new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Bigint, Value = value }; + cmd.Parameters.Add(p); + return p; + } + + /// Appends a SYMBOL/VARCHAR parameter (varchar); null binds as NULL. + /// Target command. + /// The string. + /// The added parameter. + public static NpgsqlParameter BindText(NpgsqlCommand cmd, string? value) + { + var p = new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Varchar, Value = (object?)value ?? DBNull.Value }; + cmd.Parameters.Add(p); + return p; + } + + /// Reads a TIMESTAMP column as UTC (QuestDB returns Kind=Unspecified holding UTC wall clock). + /// Reader positioned on a row. + /// Column ordinal. + /// The timestamp with Kind=Utc. + public static DateTime ReadUtc(NpgsqlDataReader r, int i) => + DateTime.SpecifyKind(r.GetDateTime(i), DateTimeKind.Utc); + + /// Reads a BYTE column (QuestDB returns smallint); NULL (never stored) reads as 0 (Empty). + /// Reader positioned on a row. + /// Column ordinal. + /// The byte value. + public static byte ReadKind(NpgsqlDataReader r, int i) => r.IsDBNull(i) ? (byte)0 : (byte)r.GetInt16(i); + + // ---- error classification -------------------------------------------------------------------------------------- + + /// + /// True for a transport-level or timeout failure worth retrying. A is a + /// server-side rejection (QuestDB reports every one as SQLSTATE 00000), so it is never transient here; decide + /// those by message text. + /// + /// The failure. + /// Whether the failure is transient. + public static bool IsTransient(Exception ex) + { + for (var e = ex; e is not null; e = e.InnerException) + { + if (e is PostgresException or OperationCanceledException) return false; + if (e is NpgsqlException { IsTransient: true } or SocketException or IOException or TimeoutException) return true; + } + return false; + } + + /// True when a server error says the object already exists (message text; SQLSTATE is always 00000). + /// The failure. + /// Whether it is an "already exists" rejection. + public static bool IsAlreadyExists(Exception ex) => + ex is PostgresException pg && pg.MessageText.Contains("already exists", StringComparison.OrdinalIgnoreCase); + + /// Whether a table exists, per tables(). + /// Open connection. + /// Full table name. + /// Cancellation. + /// True when listed. + public async Task TableExistsAsync(NpgsqlConnection conn, string table, CancellationToken ct) + { + await using var cmd = CreateCommand(conn, "SELECT count() FROM tables() WHERE table_name = $1"); + BindText(cmd, table); + return Convert.ToInt64(await cmd.ExecuteScalarAsync(ct).ConfigureAwait(false)) > 0; + } +} diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianOptions.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianOptions.cs new file mode 100644 index 00000000..c7d71be2 --- /dev/null +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianOptions.cs @@ -0,0 +1,57 @@ +using System.Text.RegularExpressions; + +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb; + +/// Options for the QuestDB historian backend (the ServerHistorian:QuestDb config block). +public sealed partial class QuestDbHistorianOptions +{ + /// + /// Required. An Npgsql connection string for QuestDB's PG-wire port, e.g. + /// Host=questdb;Port=8812;Username=admin;Password=quest;Database=qdb;ServerCompatibilityMode=NoTypeLoading. + /// + public string ConnectionString { get; set; } = ""; + + /// Prefix of the three tables (QuestDB has no schemas). Spliced into DDL/DML, so it is regex-validated. + public string TablePrefix { get; set; } = "hist_"; + + /// Retention in days applied as a TTL to the sample and event tables; null keeps data forever. + public int? RetentionDays { get; set; } + + /// Per-command timeout. Must be positive. + public TimeSpan CommandTimeout { get; set; } = TimeSpan.FromSeconds(30); + + /// + /// true creates the tables (and applies the TTL) on first use; false only verifies they exist and + /// fails every historian call with HistorianSchemaMissingException otherwise. + /// + public bool AutoCreateSchema { get; set; } = true; + + /// How long a write waits for its rows to become visible to queries (WAL apply). 0 disables the wait. + public TimeSpan VisibilityTimeout { get; set; } = TimeSpan.FromSeconds(5); + + [GeneratedRegex("^[A-Za-z0-9_]+$")] + private static partial Regex PrefixRegex(); + + /// True when is safe to splice into a table name. + /// Candidate prefix. + /// Whether it matches ^[A-Za-z0-9_]+$. + public static bool IsValidPrefix(string? prefix) => prefix is not null && PrefixRegex().IsMatch(prefix); + + /// Validates the options without touching QuestDB. + /// One message per problem (each names its ServerHistorian:QuestDb:<Key>); empty when valid. + public IReadOnlyList Validate() + { + var errors = new List(); + if (string.IsNullOrWhiteSpace(ConnectionString)) + errors.Add("ServerHistorian:QuestDb:ConnectionString is required."); + if (!IsValidPrefix(TablePrefix)) + errors.Add("ServerHistorian:QuestDb:TablePrefix must match ^[A-Za-z0-9_]+$."); + if (RetentionDays is < 1) + errors.Add("ServerHistorian:QuestDb:RetentionDays must be at least 1 when set."); + if (CommandTimeout <= TimeSpan.Zero) + errors.Add("ServerHistorian:QuestDb:CommandTimeout must be greater than zero."); + if (VisibilityTimeout < TimeSpan.Zero) + errors.Add("ServerHistorian:QuestDb:VisibilityTimeout must not be negative (0 disables the wait)."); + return errors; + } +} diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbSchemaBootstrapper.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbSchemaBootstrapper.cs new file mode 100644 index 00000000..45d753f0 --- /dev/null +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbSchemaBootstrapper.cs @@ -0,0 +1,143 @@ +using Microsoft.Extensions.Logging; +using Microsoft.Extensions.Logging.Abstractions; +using Npgsql; +using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common; + +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb; + +/// +/// Lazily creates the three tables once per instance (CREATE TABLE IF NOT EXISTS), then reconciles the TTL +/// of the sample and event tables with . +/// +/// Concurrency: two nodes starting together are safe. If a CREATE still fails with a server error +/// whose message says already exists, tables() is re-checked and the error is swallowed only when the +/// table is really there; anything else is rethrown (SQLSTATE is always 00000, so it cannot classify). +/// TTL (RC3): tables() normalises units (7 days reads back as 1 WEEK) so values are compared +/// in hours, and an ALTER becomes visible only after the WAL applies it, so the bootstrapper never re-reads to verify +/// its own ALTER. A boot whose TTL already matches issues no ALTER. +/// Failure: not latched. Within the last failure is +/// re-thrown without touching QuestDB. +/// AutoCreateSchema=false only verifies the tables exist (no TTL changes) and throws +/// otherwise. +/// +/// +internal sealed class QuestDbSchemaBootstrapper +{ + private readonly QuestDbConnection _conn; + private readonly QuestDbHistorianOptions _options; + private readonly ILogger _log; + private readonly SemaphoreSlim _gate = new(1, 1); + private readonly BootstrapFailureCache _failure; // guarded by _gate + private volatile bool _ready; + + /// Creates the bootstrapper. + /// Connection helper. + /// Validated options. + /// Optional logger. + /// Clock for the failure cache; system by default. + public QuestDbSchemaBootstrapper(QuestDbConnection conn, QuestDbHistorianOptions options, ILogger? log = null, + TimeProvider? timeProvider = null) + { + _conn = conn; + _options = options; + _log = log ?? NullLogger.Instance; + _failure = new BootstrapFailureCache(timeProvider ?? TimeProvider.System, BootstrapFailureCache.DefaultDuration); + } + + /// Ensures the schema is ready; no-op after the first success. + /// Cancellation. + /// A task completing when the tables are usable. + /// AutoCreateSchema is false and a table is absent. + public async Task EnsureAsync(CancellationToken ct) + { + if (_ready) return; + await _gate.WaitAsync(ct).ConfigureAwait(false); + try + { + if (_ready) return; + _failure.ThrowIfCached(); + try + { + await using var conn = await _conn.OpenAsync(ct).ConfigureAwait(false); + if (_options.AutoCreateSchema) + { + await CreateTablesAsync(conn, ct).ConfigureAwait(false); + await ReconcileTtlAsync(conn, ct).ConfigureAwait(false); + } + else + { + await VerifyAsync(conn, ct).ConfigureAwait(false); + } + _ready = true; + _failure.Clear(); + } + catch (Exception ex) when (ex is not OperationCanceledException || !ct.IsCancellationRequested) + { + _failure.Record(ex); + _log.LogWarning("QuestDB historian schema bootstrap (prefix '{Prefix}') failed ({Exception}: {Message}); " + + "historian calls fail fast for {Seconds} s before retrying.", + _conn.TablePrefix, ex.GetType().Name, ex.Message, BootstrapFailureCache.DefaultDuration.TotalSeconds); + throw; + } + } + finally + { + _gate.Release(); + } + } + + private async Task VerifyAsync(NpgsqlConnection conn, CancellationToken ct) + { + foreach (var table in QuestDbTables.All(_conn.TablePrefix)) + { + if (!await _conn.TableExistsAsync(conn, table, ct).ConfigureAwait(false)) + throw new HistorianSchemaMissingException( + $"QuestDB historian table '{table}' is missing and AutoCreateSchema is false. " + + "Create the tables (see docs/Historian.md) or enable AutoCreateSchema."); + } + } + + private async Task CreateTablesAsync(NpgsqlConnection conn, CancellationToken ct) + { + var statements = QuestDbTables.CreateAll(_conn.TablePrefix, _options.RetentionDays); + var tables = QuestDbTables.All(_conn.TablePrefix); + for (var i = 0; i < statements.Length; i++) + { + try + { + await using var cmd = _conn.CreateCommand(conn, statements[i]); + await cmd.ExecuteNonQueryAsync(ct).ConfigureAwait(false); + } + catch (PostgresException ex) when (QuestDbConnection.IsAlreadyExists(ex)) + { + // A concurrent creator won the race - swallow ONLY if the table is really there. + if (!await _conn.TableExistsAsync(conn, tables[i], ct).ConfigureAwait(false)) throw; + _log.LogDebug("QuestDB table {Table} was created concurrently.", tables[i]); + } + } + } + + private async Task ReconcileTtlAsync(NpgsqlConnection conn, CancellationToken ct) + { + var wantHours = _options.RetentionDays is { } d ? d * 24L : 0L; + foreach (var table in QuestDbTables.Expiring(_conn.TablePrefix)) + { + long haveHours; + await using (var cmd = _conn.CreateCommand(conn, "SELECT ttlValue, ttlUnit FROM tables() WHERE table_name = $1")) + { + QuestDbConnection.BindText(cmd, table); + await using var r = await cmd.ExecuteReaderAsync(ct).ConfigureAwait(false); + if (!await r.ReadAsync(ct).ConfigureAwait(false)) continue; // vanished (dropped between create and here) + var value = r.IsDBNull(0) ? 0L : Convert.ToInt64(r.GetValue(0)); + var unit = r.IsDBNull(1) ? null : r.GetString(1); + haveHours = QuestDbTables.TtlHours(value, unit); + } + if (haveHours == wantHours) continue; + + // The ALTER is a WAL transaction: tables() keeps the old value until it applies, so do NOT re-read to verify. + await using (var alter = _conn.CreateCommand(conn, QuestDbTables.AlterTtl(table, _options.RetentionDays))) + await alter.ExecuteNonQueryAsync(ct).ConfigureAwait(false); + _log.LogInformation("Applied TTL {Days} day(s) to QuestDB table {Table}.", _options.RetentionDays, table); + } + } +} diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbTables.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbTables.cs new file mode 100644 index 00000000..c7a1ad21 --- /dev/null +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbTables.cs @@ -0,0 +1,83 @@ +using System.Globalization; + +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb; + +/// +/// Table names and DDL for the QuestDB backend (T0-confirmed against QuestDB 10.0.1). The TTL clause sits +/// between PARTITION BY and WAL and is an inlined integer (QuestDB cannot parameterise it). +/// Prefixes are regex-validated by the caller; these builders also refuse an invalid one. +/// +internal static class QuestDbTables +{ + /// Base name of the sample table (prefix + this). + public const string SampleBase = "sample"; + /// Base name of the event table. + public const string EventBase = "event"; + /// Base name of the tag catalog table. + public const string TagBase = "tag"; + + /// The fixed designated timestamp of every tag row, which makes the dedup key an upsert by name. + public static readonly DateTime TagEpoch = DateTime.UnixEpoch; + + /// The prefixed sample table name. + public static string Sample(string prefix) => Name(prefix, SampleBase); + /// The prefixed event table name. + public static string Event(string prefix) => Name(prefix, EventBase); + /// The prefixed tag table name. + public static string Tag(string prefix) => Name(prefix, TagBase); + + /// All three table names in creation order. + public static string[] All(string prefix) => [Sample(prefix), Event(prefix), Tag(prefix)]; + + /// The tables that carry a TTL (sample and event; the tag catalog never expires). + public static string[] Expiring(string prefix) => [Sample(prefix), Event(prefix)]; + + /// CREATE TABLE IF NOT EXISTS for the sample table. + public static string CreateSample(string prefix, int? retentionDays) => + $"CREATE TABLE IF NOT EXISTS {Sample(prefix)} (ts TIMESTAMP, tag SYMBOL, kind BYTE, num_value DOUBLE, " + + "int_value LONG, str_value VARCHAR, status LONG, deleted BOOLEAN, server_ts TIMESTAMP) " + + $"TIMESTAMP(ts) PARTITION BY DAY{Ttl(retentionDays)} WAL DEDUP UPSERT KEYS(ts, tag)"; + + /// CREATE TABLE IF NOT EXISTS for the event table. + public static string CreateEvent(string prefix, int? retentionDays) => + $"CREATE TABLE IF NOT EXISTS {Event(prefix)} (ts TIMESTAMP, event_id VARCHAR, source SYMBOL, alarm_id VARCHAR, " + + "alarm_name VARCHAR, alarm_type_name VARCHAR, severity INT, event_kind VARCHAR, message VARCHAR, " + + "user_name VARCHAR, comment VARCHAR, received_utc TIMESTAMP) " + + $"TIMESTAMP(ts) PARTITION BY DAY{Ttl(retentionDays)} WAL DEDUP UPSERT KEYS(ts, event_id)"; + + /// CREATE TABLE IF NOT EXISTS for the tag catalog (YEAR partitions, no TTL ever). + public static string CreateTag(string prefix) => + $"CREATE TABLE IF NOT EXISTS {Tag(prefix)} (ts TIMESTAMP, name SYMBOL, data_type BYTE, eng_unit VARCHAR, " + + "description VARCHAR, updated_utc TIMESTAMP) TIMESTAMP(ts) PARTITION BY YEAR WAL DEDUP UPSERT KEYS(ts, name)"; + + /// The three CREATE statements in creation order. + public static string[] CreateAll(string prefix, int? retentionDays) => + [CreateSample(prefix, retentionDays), CreateEvent(prefix, retentionDays), CreateTag(prefix)]; + + /// ALTER TABLE t SET TTL n DAYS; null removes the TTL (SET TTL 0 DAYS). + public static string AlterTtl(string table, int? retentionDays) => + $"ALTER TABLE {table} SET TTL {(retentionDays ?? 0).ToString(CultureInfo.InvariantCulture)} DAYS"; + + /// + /// Converts a tables() ttlValue/ttlUnit pair to hours. QuestDB normalises the unit (7 days + /// reads back as 1 WEEK), so equality is judged in hours. An unrecognised unit returns -1 (never equal to + /// a wanted value, so the bootstrapper re-applies the TTL). + /// + public static long TtlHours(long value, string? unit) => (unit ?? "").ToUpperInvariant() switch + { + "HOUR" => value, + "DAY" => value * 24, + "WEEK" => value * 168, + _ => value == 0 ? 0 : -1, + }; + + private static string Ttl(int? days) => + days is { } d ? $" TTL {d.ToString(CultureInfo.InvariantCulture)} DAYS" : ""; + + private static string Name(string prefix, string baseName) + { + if (!QuestDbHistorianOptions.IsValidPrefix(prefix)) + throw new ArgumentException("Invalid table prefix.", nameof(prefix)); + return prefix + baseName; + } +} diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbTagCatalog.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbTagCatalog.cs new file mode 100644 index 00000000..341ba89d --- /dev/null +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbTagCatalog.cs @@ -0,0 +1,82 @@ +using System.Collections.Concurrent; +using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common; + +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb; + +/// +/// The only "is this tag known" authority for the QuestDB backend (R7): a row in the tag table (fixed designated +/// timestamp 1970-01-01, deduplicated by name). Caches positive results only. Both methods run the schema +/// bootstrap first. Names longer than are never truncated: +/// answers false and throws. +/// inserts only when the tag is absent, because the table's dedup key replaces a +/// whole row: re-inserting a known tag would erase the engineering unit and description the provisioner wrote. +/// +internal sealed class QuestDbTagCatalog +{ + private readonly QuestDbConnection _conn; + private readonly QuestDbSchemaBootstrapper _bootstrap; + private readonly QuestDbWalVisibility _wal; + private readonly TimeProvider _clock; + private readonly ConcurrentDictionary _known = new(StringComparer.Ordinal); + + /// Creates the catalog. + /// Connection helper. + /// Schema bootstrapper. + /// WAL visibility wait. + /// Clock for updated_utc; system by default. + public QuestDbTagCatalog(QuestDbConnection conn, QuestDbSchemaBootstrapper bootstrap, QuestDbWalVisibility wal, + TimeProvider? clock = null) + { + _conn = conn; + _bootstrap = bootstrap; + _wal = wal; + _clock = clock ?? TimeProvider.System; + } + + /// Read-only lookup; never creates a row. + /// Historian tag name. + /// Cancellation. + /// True when the tag is in the catalog; false for an unknown or over-long name. + public async Task ExistsAsync(string name, CancellationToken ct) + { + if (!HistorianTagNames.IsValid(name)) return false; + if (_known.ContainsKey(name)) return true; + await _bootstrap.EnsureAsync(ct).ConfigureAwait(false); + await using var conn = await _conn.OpenAsync(ct).ConfigureAwait(false); + await using var cmd = _conn.CreateCommand(conn, $"SELECT count() FROM {_conn.TagTable} WHERE name = $1"); + QuestDbConnection.BindText(cmd, name); + var found = Convert.ToInt64(await cmd.ExecuteScalarAsync(ct).ConfigureAwait(false)) > 0; + if (found) _known[name] = 0; + return found; + } + + /// Registers the tag when absent (no-op when present) and waits for the row to become visible. + /// Historian tag name. + /// Initial data_type (only written on create; provisioning corrects it later). + /// Cancellation. + /// A task completing when the tag is registered and visible. + /// The name is longer than . + public async Task EnsureAsync(string name, byte dataType, CancellationToken ct) + { + if (!HistorianTagNames.IsValid(name)) + throw new ArgumentException($"Historian tag name exceeds {HistorianTagNames.MaxLength} characters.", nameof(name)); + if (await ExistsAsync(name, ct).ConfigureAwait(false)) return; + + await using (var conn = await _conn.OpenAsync(ct).ConfigureAwait(false)) + { + await using var cmd = _conn.CreateCommand(conn, + $"INSERT INTO {_conn.TagTable} (ts, name, data_type, updated_utc) VALUES ($1, $2, $3, $4)"); + QuestDbConnection.BindTimestamp(cmd, QuestDbTables.TagEpoch); + QuestDbConnection.BindText(cmd, name); + QuestDbConnection.BindByte(cmd, dataType); + QuestDbConnection.BindTimestamp(cmd, _clock.GetUtcNow().UtcDateTime); + await cmd.ExecuteNonQueryAsync(ct).ConfigureAwait(false); + } + await _wal.WaitAsync(_conn.TagTable, ct).ConfigureAwait(false); + _known[name] = 0; + } + + /// Marks a tag as known after another component (the provisioner) has written its row and waited. + /// Historian tag name. + internal void MarkKnown(string name) => _known[name] = 0; +} diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbWalVisibility.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbWalVisibility.cs new file mode 100644 index 00000000..578ff3bd --- /dev/null +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbWalVisibility.cs @@ -0,0 +1,92 @@ +using System.Diagnostics; +using Microsoft.Extensions.Logging; +using Microsoft.Extensions.Logging.Abstractions; + +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb; + +/// +/// The read-your-writes wait (R5, T0 Q4). QuestDB applies WAL commits asynchronously, so a row just inserted may +/// not be visible to the next query. After its INSERT(s) a writer calls , which captures +/// sequencerTxn from wal_tables() (every autocommitted INSERT is one sequencer transaction; a concurrent +/// writer can only raise the target) and polls writerTxn every ~10 ms until it reaches the target, the table +/// is suspended, or elapses. +/// +internal sealed class QuestDbWalVisibility +{ + private static readonly TimeSpan PollInterval = TimeSpan.FromMilliseconds(10); + + private readonly QuestDbConnection _conn; + private readonly TimeSpan _timeout; + private readonly ILogger _log; + + /// Creates the waiter. + /// Connection helper. + /// Maximum wait; (or less) disables the wait. + /// Optional logger. + public QuestDbWalVisibility(QuestDbConnection conn, TimeSpan timeout, ILogger? log = null) + { + _conn = conn; + _timeout = timeout; + _log = log ?? NullLogger.Instance; + } + + /// True when the wait is disabled (VisibilityTimeout is 0). + public bool Disabled => _timeout <= TimeSpan.Zero; + + /// True when the applied transaction has caught up with the target, or the table cannot progress. + /// Last applied transaction. + /// The sequencer transaction captured after the INSERT. + /// Whether polling can stop because the rows are visible. + internal static bool Reached(long writerTxn, long targetTxn) => writerTxn >= targetTxn; + + /// Waits until everything this caller inserted into is visible to queries. + /// Full table name written to. + /// Cancellation. + /// + /// True when visible (or the wait is disabled, or the table is not a WAL table); false when the table is suspended + /// or the timeout elapsed (a Warning is logged; the rows are still committed and become visible later). + /// + public async Task WaitAsync(string table, CancellationToken ct) + { + if (Disabled) return true; + var sw = Stopwatch.StartNew(); + await using var conn = await _conn.OpenAsync(ct).ConfigureAwait(false); + + long target; + await using (var cmd = _conn.CreateCommand(conn, "SELECT sequencerTxn FROM wal_tables() WHERE name = $1")) + { + QuestDbConnection.BindText(cmd, table); + var o = await cmd.ExecuteScalarAsync(ct).ConfigureAwait(false); + if (o is null or DBNull) return true; // not a WAL table: writes are immediately visible + target = Convert.ToInt64(o); + } + + while (true) + { + long writer; + bool suspended; + await using (var cmd = _conn.CreateCommand(conn, "SELECT writerTxn, suspended FROM wal_tables() WHERE name = $1")) + { + QuestDbConnection.BindText(cmd, table); + await using var r = await cmd.ExecuteReaderAsync(ct).ConfigureAwait(false); + if (!await r.ReadAsync(ct).ConfigureAwait(false)) return true; + writer = r.GetInt64(0); + suspended = r.GetBoolean(1); + } + if (Reached(writer, target)) return true; + if (suspended) + { + _log.LogWarning("QuestDB table {Table} is suspended (writerTxn {Writer} < sequencerTxn {Target}); not waiting.", + table, writer, target); + return false; + } + if (sw.Elapsed >= _timeout) + { + _log.LogWarning("QuestDB table {Table} did not become visible within {Timeout} ms (writerTxn {Writer} < sequencerTxn {Target}).", + table, _timeout.TotalMilliseconds, writer, target); + return false; + } + await Task.Delay(PollInterval, ct).ConfigureAwait(false); + } + } +} diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.csproj b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.csproj new file mode 100644 index 00000000..0a2487e9 --- /dev/null +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.csproj @@ -0,0 +1,30 @@ + + + + true + true + $(NoWarn);CS1591 + AnyCPU;x64 + ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb + + + + + + + + + + + + + + + + + + + + + + diff --git a/tests/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions.Tests/ZB.MOM.WW.OtOpcUa.Core.Abstractions.Tests.csproj b/tests/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions.Tests/ZB.MOM.WW.OtOpcUa.Core.Abstractions.Tests.csproj index 6cae72d6..4061cda3 100644 --- a/tests/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions.Tests/ZB.MOM.WW.OtOpcUa.Core.Abstractions.Tests.csproj +++ b/tests/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions.Tests/ZB.MOM.WW.OtOpcUa.Core.Abstractions.Tests.csproj @@ -44,6 +44,7 @@ puts it in StatusCodeParityTests' scope. --> + diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/BootstrapTests.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/BootstrapTests.cs new file mode 100644 index 00000000..cae1231e --- /dev/null +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/BootstrapTests.cs @@ -0,0 +1,241 @@ +using Npgsql; +using Shouldly; +using Xunit; +using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common; + +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests; + +[Collection(QuestDbCollection.Name)] +public sealed class BootstrapTests(QuestDbFixture fx) +{ + private (QuestDbConnection Conn, QuestDbSchemaBootstrapper Boot, QuestDbHistorianOptions Opts) Make( + string prefix, int? retention = null, bool autoCreate = true) + { + var o = fx.Options(prefix, retention, autoCreate); + var c = new QuestDbConnection(o); + return (c, new QuestDbSchemaBootstrapper(c, o), o); + } + + private static async Task> TablesAsync(QuestDbConnection c, string prefix) + { + await using var conn = await c.OpenAsync(default); + await using var cmd = c.CreateCommand(conn, "SELECT table_name FROM tables() WHERE starts_with(table_name, $1) ORDER BY table_name"); + QuestDbConnection.BindText(cmd, prefix); + var names = new List(); + await using var r = await cmd.ExecuteReaderAsync(); + while (await r.ReadAsync()) names.Add(r.GetString(0)); + return names; + } + + private static async Task<(long Value, string Unit)> TtlAsync(QuestDbConnection c, string table) + { + await using var conn = await c.OpenAsync(default); + await using var cmd = c.CreateCommand(conn, "SELECT ttlValue, ttlUnit FROM tables() WHERE table_name = $1"); + QuestDbConnection.BindText(cmd, table); + await using var r = await cmd.ExecuteReaderAsync(); + (await r.ReadAsync()).ShouldBeTrue(); + return (Convert.ToInt64(r.GetValue(0)), r.GetString(1)); + } + + private static async Task SequencerTxnAsync(QuestDbConnection c, string table) + { + await using var conn = await c.OpenAsync(default); + await using var cmd = c.CreateCommand(conn, "SELECT sequencerTxn FROM wal_tables() WHERE name = $1"); + QuestDbConnection.BindText(cmd, table); + return Convert.ToInt64(await cmd.ExecuteScalarAsync()); + } + + [Fact] + public async Task Tables_are_created() + { + fx.SkipIfUnavailable(); + var p = fx.UniquePrefix(); + var (c, boot, _) = Make(p); + await boot.EnsureAsync(default); + (await TablesAsync(c, p)).ShouldBe([p + "event", p + "sample", p + "tag"]); + } + + [Fact] + public async Task A_second_run_is_a_noop_and_issues_no_alter() + { + fx.SkipIfUnavailable(); + var p = fx.UniquePrefix(); + var (c, boot, _) = Make(p, retention: 7); + await boot.EnsureAsync(default); + var before = await SequencerTxnAsync(c, p + "sample"); + + var (c2, boot2, _) = Make(p, retention: 7); // a fresh instance on a table whose TTL already matches (stored as 1 WEEK) + await boot2.EnsureAsync(default); + await Task.Delay(300); // an ALTER would have been applied by now + (await SequencerTxnAsync(c2, p + "sample")).ShouldBe(before); + (await SequencerTxnAsync(c2, p + "event")).ShouldBe(await SequencerTxnAsync(c, p + "event")); + (await TablesAsync(c2, p)).Count.ShouldBe(3); + } + + [Fact] + public async Task Two_concurrent_bootstrappers_both_succeed() + { + fx.SkipIfUnavailable(); + for (var round = 0; round < 5; round++) + { + var p = fx.UniquePrefix(); + var a = Make(p, retention: 7); + var b = Make(p, retention: 7); + await Task.WhenAll( + Task.Run(() => a.Boot.EnsureAsync(default)), + Task.Run(() => b.Boot.EnsureAsync(default))); + (await TablesAsync(a.Conn, p)).Count.ShouldBe(3); + } + } + + [Fact] + public async Task Ttl_is_in_metadata_when_retention_is_7_days_and_absent_when_unset() + { + fx.SkipIfUnavailable(); + var withTtl = fx.UniquePrefix(); + var (c1, b1, _) = Make(withTtl, retention: 7); + await b1.EnsureAsync(default); + QuestDbTables.TtlHours((await TtlAsync(c1, withTtl + "sample")).Value, (await TtlAsync(c1, withTtl + "sample")).Unit).ShouldBe(168); + QuestDbTables.TtlHours((await TtlAsync(c1, withTtl + "event")).Value, (await TtlAsync(c1, withTtl + "event")).Unit).ShouldBe(168); + (await TtlAsync(c1, withTtl + "tag")).Value.ShouldBe(0); // the tag catalog never expires + + var noTtl = fx.UniquePrefix(); + var (c2, b2, _) = Make(noTtl); + await b2.EnsureAsync(default); + (await TtlAsync(c2, noTtl + "sample")).Value.ShouldBe(0); + (await TtlAsync(c2, noTtl + "event")).Value.ShouldBe(0); + } + + [Fact] + public async Task A_changed_retention_is_reapplied_and_removal_clears_it() + { + fx.SkipIfUnavailable(); + var p = fx.UniquePrefix(); + var (c, b7, _) = Make(p, retention: 7); + await b7.EnsureAsync(default); + + await Make(p, retention: 30).Boot.EnsureAsync(default); + await AwaitHoursAsync(c, p + "sample", 720); + await AwaitHoursAsync(c, p + "event", 720); + + await Make(p).Boot.EnsureAsync(default); // unset => SET TTL 0 + await AwaitHoursAsync(c, p + "sample", 0); + } + + private static async Task AwaitHoursAsync(QuestDbConnection c, string table, long hours) + { + var deadline = DateTime.UtcNow.AddSeconds(10); + long got; + do + { + var (v, u) = await TtlAsync(c, table); + got = QuestDbTables.TtlHours(v, u); + if (got == hours) return; + await Task.Delay(50); // the ALTER is a WAL transaction: visible only once applied + } while (DateTime.UtcNow < deadline); + got.ShouldBe(hours); + } + + [Fact] + public async Task AutoCreateSchema_false_against_missing_tables_throws_HistorianSchemaMissingException() + { + fx.SkipIfUnavailable(); + var (_, boot, _) = Make(fx.UniquePrefix(), autoCreate: false); + var ex = await Should.ThrowAsync(() => boot.EnsureAsync(default)); + ex.Message.ShouldContain("AutoCreateSchema"); + } + + [Fact] + public async Task AutoCreateSchema_false_passes_once_the_tables_exist() + { + fx.SkipIfUnavailable(); + var p = fx.UniquePrefix(); + await Make(p).Boot.EnsureAsync(default); + await Make(p, autoCreate: false).Boot.EnsureAsync(default); + } + + [Fact] + public async Task ExistsAsync_is_false_for_an_unknown_tag_and_true_after_EnsureAsync() + { + fx.SkipIfUnavailable(); + var p = fx.UniquePrefix(); + var (c, boot, o) = Make(p); + var catalog = new QuestDbTagCatalog(c, boot, new QuestDbWalVisibility(c, o.VisibilityTimeout)); + + (await catalog.ExistsAsync("Plant.Pump1.Speed", default)).ShouldBeFalse(); + await catalog.EnsureAsync("Plant.Pump1.Speed", 3, default); + (await catalog.ExistsAsync("Plant.Pump1.Speed", default)).ShouldBeTrue(); + // A fresh catalog (no cache) sees it through the table, and a different name is still unknown. + var fresh = new QuestDbTagCatalog(c, boot, new QuestDbWalVisibility(c, o.VisibilityTimeout)); + (await fresh.ExistsAsync("Plant.Pump1.Speed", default)).ShouldBeTrue(); + (await fresh.ExistsAsync("Plant.Pump1.Other", default)).ShouldBeFalse(); + await catalog.EnsureAsync("Plant.Pump1.Speed", 3, default); // idempotent + } + + [Fact] + public async Task An_over_long_tag_name_is_unknown_and_cannot_be_ensured() + { + fx.SkipIfUnavailable(); + var p = fx.UniquePrefix(); + var (c, boot, o) = Make(p); + var catalog = new QuestDbTagCatalog(c, boot, new QuestDbWalVisibility(c, o.VisibilityTimeout)); + var name = new string('x', HistorianTagNames.MaxLength + 1); + (await catalog.ExistsAsync(name, default)).ShouldBeFalse(); + await Should.ThrowAsync(() => catalog.EnsureAsync(name, 3, default)); + } + + [Fact] + public async Task Wait_returns_once_writerTxn_reaches_the_captured_sequencerTxn() + { + fx.SkipIfUnavailable(); + var p = fx.UniquePrefix(); + var (c, boot, o) = Make(p); + await boot.EnsureAsync(default); + var wal = new QuestDbWalVisibility(c, TimeSpan.FromSeconds(5)); + for (var i = 0; i < 20; i++) + { + var ts = new DateTime(2026, 3, 1, 0, 0, i, DateTimeKind.Utc); + await using (var conn = await c.OpenAsync(default)) + { + await using var cmd = c.CreateCommand(conn, + $"INSERT INTO {c.SampleTable} (ts, tag, kind, num_value, status, deleted) VALUES ($1, $2, $3, $4, $5, false)"); + QuestDbConnection.BindTimestamp(cmd, ts); + QuestDbConnection.BindText(cmd, "wal-tag"); + QuestDbConnection.BindByte(cmd, 3); + cmd.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlTypes.NpgsqlDbType.Double, Value = (double)i }); + QuestDbConnection.BindLong(cmd, 0); + await cmd.ExecuteNonQueryAsync(); + } + (await wal.WaitAsync(c.SampleTable, default)).ShouldBeTrue(); + // After the wait the row is visible on the very next query (the T0 gap was 7/20 without it). + await using var q = await c.OpenAsync(default); + await using var count = c.CreateCommand(q, $"SELECT count() FROM {c.SampleTable} WHERE tag = $1"); + QuestDbConnection.BindText(count, "wal-tag"); + Convert.ToInt64(await count.ExecuteScalarAsync()).ShouldBe(i + 1); + } + } + + [Fact] + public async Task Timeout_zero_skips_the_wait() + { + fx.SkipIfUnavailable(); + var p = fx.UniquePrefix(); + var (c, boot, _) = Make(p); + await boot.EnsureAsync(default); + var wal = new QuestDbWalVisibility(c, TimeSpan.Zero); + var sw = System.Diagnostics.Stopwatch.StartNew(); + (await wal.WaitAsync(c.SampleTable, default)).ShouldBeTrue(); + sw.ElapsedMilliseconds.ShouldBeLessThan(50); // no round-trip at all + } + + [Fact] + public async Task Waiting_on_a_table_that_is_not_a_WAL_table_returns_immediately() + { + fx.SkipIfUnavailable(); + var p = fx.UniquePrefix(); + var (c, boot, _) = Make(p); + await boot.EnsureAsync(default); + var wal = new QuestDbWalVisibility(c, TimeSpan.FromSeconds(5)); + (await wal.WaitAsync(p + "no_such_table", default)).ShouldBeTrue(); + } +} diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/QuestDbFixture.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/QuestDbFixture.cs new file mode 100644 index 00000000..dcaba660 --- /dev/null +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/QuestDbFixture.cs @@ -0,0 +1,106 @@ +using System.Collections.Concurrent; +using Npgsql; +using Xunit; + +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests; + +/// +/// Env-gated QuestDB for the historian integration tests. HISTORIAN_QUESTDB_TEST_CONNECTION absent ⇒ +/// is set, nothing connects and every test calls Assert.Skip. Present ⇒ tests isolate +/// from each other with ; every prefix handed out is dropped on dispose (and a test may +/// drop its own early with ). Never touches a table outside those prefixes. +/// +public sealed class QuestDbFixture : IAsyncLifetime +{ + /// Env var holding an Npgsql connection string for QuestDB's PG-wire port. + public const string ConnectionEnvVar = "HISTORIAN_QUESTDB_TEST_CONNECTION"; + + private readonly ConcurrentBag _prefixes = new(); + + /// Why tests must skip, or null when QuestDB is available. + public string? SkipReason { get; private set; } + + /// The Npgsql connection string. Valid only when is null. + public string ConnectionString { get; private set; } = ""; + + /// Skips the calling test when the gate is closed. + public void SkipIfUnavailable() + { + if (SkipReason is not null) Assert.Skip(SkipReason); + } + + /// A fresh, valid table prefix ("t" + 12 hex + "_") that nothing uses yet; dropped on dispose. + /// The prefix. + public string UniquePrefix() + { + var p = "t" + Guid.NewGuid().ToString("N")[..12] + "_"; + _prefixes.Add(p); + return p; + } + + /// Options targeting this fixture's QuestDB under . + /// Table prefix (normally from ). + /// Optional TTL. + /// AutoCreateSchema. + /// Valid options. + public QuestDbHistorianOptions Options(string prefix, int? retentionDays = null, bool autoCreate = true) => new() + { + ConnectionString = ConnectionString, + TablePrefix = prefix, + RetentionDays = retentionDays, + AutoCreateSchema = autoCreate, + }; + + /// Drops every table whose name starts with (found through tables()). + /// Table prefix. + /// A task completing when they are gone. + public async Task DropPrefixAsync(string prefix) + { + if (SkipReason is not null) return; + if (!QuestDbHistorianOptions.IsValidPrefix(prefix)) throw new ArgumentException("Invalid prefix.", nameof(prefix)); + await using var ds = NpgsqlDataSource.Create(ConnectionString); + await using var conn = await ds.OpenConnectionAsync().ConfigureAwait(false); + var names = new List(); + await using (var cmd = conn.CreateCommand()) + { + cmd.CommandText = "SELECT table_name FROM tables() WHERE starts_with(table_name, $1)"; // never LIKE: '_' is a wildcard + cmd.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlTypes.NpgsqlDbType.Varchar, Value = prefix }); + await using var r = await cmd.ExecuteReaderAsync().ConfigureAwait(false); + while (await r.ReadAsync().ConfigureAwait(false)) names.Add(r.GetString(0)); + } + foreach (var n in names) + { + await using var drop = conn.CreateCommand(); + drop.CommandText = $"DROP TABLE IF EXISTS {n}"; + await drop.ExecuteNonQueryAsync().ConfigureAwait(false); + } + } + + /// + public ValueTask InitializeAsync() + { + var cs = Environment.GetEnvironmentVariable(ConnectionEnvVar); + if (string.IsNullOrWhiteSpace(cs)) + SkipReason = $"{ConnectionEnvVar} is not set; QuestDB historian integration tests skipped."; + else + ConnectionString = cs; + return ValueTask.CompletedTask; + } + + /// + public async ValueTask DisposeAsync() + { + if (SkipReason is not null) return; + foreach (var p in _prefixes.Distinct()) + await DropPrefixAsync(p).ConfigureAwait(false); + NpgsqlConnection.ClearAllPools(); + } +} + +/// Shares one fixture across the QuestDB integration tests. +[CollectionDefinition(Name)] +public sealed class QuestDbCollection : ICollectionFixture +{ + /// Collection name. + public const string Name = "QuestDb"; +} diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests.csproj b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests.csproj new file mode 100644 index 00000000..c34ce291 --- /dev/null +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests.csproj @@ -0,0 +1,33 @@ + + + + net10.0 + enable + enable + false + true + ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests + $(NoWarn);xUnit1051 + + + + + + + + all + runtime; build; native; contentfiles; analyzers; buildtransitive + + + + + + + + + + + + + + diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests/QuestDbConnectionTests.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests/QuestDbConnectionTests.cs new file mode 100644 index 00000000..0f39bf6e --- /dev/null +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests/QuestDbConnectionTests.cs @@ -0,0 +1,140 @@ +using System.Net.Sockets; +using Npgsql; +using NpgsqlTypes; +using Shouldly; +using Xunit; + +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests; + +public sealed class QuestDbConnectionTests +{ + private static DateTime Bound(DateTime v, bool clamp = false) + { + var cmd = new NpgsqlCommand(); + var p = QuestDbConnection.BindTimestamp(cmd, v, clamp); + p.NpgsqlDbType.ShouldBe(NpgsqlDbType.TimestampTz); + var bound = (DateTime)p.Value!; + bound.Kind.ShouldBe(DateTimeKind.Utc); + return bound; + } + + [Fact] + public void Round_truncates_sub_microsecond_ticks() + { + var t = new DateTime(2026, 3, 1, 12, 0, 0, DateTimeKind.Utc).AddTicks(1234567); + QuestDbConnection.Round(t).ShouldBe(new DateTime(2026, 3, 1, 12, 0, 0, DateTimeKind.Utc).AddTicks(1234560)); + } + + [Fact] + public void Round_truncates_rather_than_rounds_before_2000() + { + var t = new DateTime(1990, 5, 5, 0, 0, 0, DateTimeKind.Utc).AddTicks(1234565); + QuestDbConnection.Round(t).Ticks.ShouldBe(t.Ticks - 5); + } + + [Fact] + public void Round_treats_Unspecified_as_UTC() + { + var u = new DateTime(2026, 3, 1, 12, 0, 0, DateTimeKind.Unspecified); + var r = QuestDbConnection.Round(u); + r.Kind.ShouldBe(DateTimeKind.Utc); + r.Ticks.ShouldBe(u.Ticks); + } + + [Fact] + public void Round_converts_Local_to_UTC() + { + var local = new DateTime(2026, 3, 1, 12, 0, 0, DateTimeKind.Local); + var r = QuestDbConnection.Round(local); + r.Kind.ShouldBe(DateTimeKind.Utc); + r.ShouldBe(local.ToUniversalTime()); + } + + [Fact] + public void Round_keeps_a_UTC_value_that_is_already_aligned() + { + var t = new DateTime(2026, 3, 1, 12, 0, 0, DateTimeKind.Utc); + QuestDbConnection.Round(t).ShouldBe(t); + } + + [Theory] + [InlineData(0)] // MinValue itself + [InlineData(5)] // truncates to MinValue + public void BindTimestamp_never_emits_MinValue(long ticks) + { + var b = Bound(new DateTime(ticks, DateTimeKind.Utc)); + b.ShouldNotBe(DateTime.MinValue); + b.Ticks.ShouldBe(10); + b.ShouldBe(new DateTime(10, DateTimeKind.Utc)); // 0001-01-01T00:00:00.000001 (T0-confirmed) + } + + [Fact] + public void BindTimestamp_never_emits_MaxValue() + { + var b = Bound(DateTime.MaxValue); + b.ShouldNotBe(DateTime.MaxValue); + b.ShouldBe(new DateTime(9999, 12, 31, 23, 59, 59, DateTimeKind.Utc).AddTicks(9999990)); // T0-confirmed bound + Bound(DateTime.MinValue).ShouldNotBe(DateTime.MinValue); + } + + [Fact] + public void BindTimestamp_never_emits_either_extreme_for_any_kind() + { + foreach (var kind in new[] { DateTimeKind.Utc, DateTimeKind.Unspecified, DateTimeKind.Local }) + foreach (var t in new[] { new DateTime(DateTime.MinValue.Ticks, kind), new DateTime(DateTime.MaxValue.Ticks, kind) }) + { + var b = Bound(t); + b.ShouldNotBe(DateTime.MinValue); + b.ShouldNotBe(DateTime.MaxValue); + } + } + + [Fact] + public void BindTimestamp_keeps_a_pre_1970_range_bound_unless_asked_to_clamp() + { + var pre = new DateTime(1960, 1, 1, 0, 0, 0, DateTimeKind.Utc); + Bound(pre).ShouldBe(pre); + Bound(pre, clamp: true).ShouldBe(DateTime.UnixEpoch); + Bound(new DateTime(2026, 1, 1, 0, 0, 0, DateTimeKind.Utc), clamp: true).Year.ShouldBe(2026); + } + + [Fact] + public void IsStorable_is_false_only_before_1970() + { + QuestDbConnection.IsStorable(DateTime.UnixEpoch).ShouldBeTrue(); + QuestDbConnection.IsStorable(DateTime.UnixEpoch.AddTicks(-1)).ShouldBeFalse(); + QuestDbConnection.IsStorable(new DateTime(1601, 1, 1, 0, 0, 0, DateTimeKind.Utc)).ShouldBeFalse(); + QuestDbConnection.IsStorable(DateTime.MinValue).ShouldBeFalse(); + QuestDbConnection.IsStorable(DateTime.MaxValue).ShouldBeTrue(); + QuestDbConnection.IsStorable(new DateTime(2026, 1, 1, 0, 0, 0, DateTimeKind.Utc)).ShouldBeTrue(); + } + + [Fact] + public void BindByte_binds_smallint() + { + var p = QuestDbConnection.BindByte(new NpgsqlCommand(), 200); + p.NpgsqlDbType.ShouldBe(NpgsqlDbType.Smallint); + p.Value.ShouldBe((short)200); + } + + [Fact] + public void IsTransient_classifies_transport_failures_not_server_rejections() + { + QuestDbConnection.IsTransient(new SocketException()).ShouldBeTrue(); + QuestDbConnection.IsTransient(new IOException("x")).ShouldBeTrue(); + QuestDbConnection.IsTransient(new TimeoutException()).ShouldBeTrue(); + QuestDbConnection.IsTransient(new NpgsqlException("x", new IOException())).ShouldBeTrue(); + QuestDbConnection.IsTransient(new InvalidOperationException("x")).ShouldBeFalse(); + QuestDbConnection.IsTransient(new OperationCanceledException()).ShouldBeFalse(); + } + + [Fact] + public void T_prefixes_the_table_base_name() + { + var c = new QuestDbConnection(new QuestDbHistorianOptions { ConnectionString = "Host=127.0.0.1;Port=1", TablePrefix = "p_" }); + c.T("sample").ShouldBe("p_sample"); + c.SampleTable.ShouldBe("p_sample"); + c.EventTable.ShouldBe("p_event"); + c.TagTable.ShouldBe("p_tag"); + } +} diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests/QuestDbHistorianOptionsTests.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests/QuestDbHistorianOptionsTests.cs new file mode 100644 index 00000000..6c37c63d --- /dev/null +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests/QuestDbHistorianOptionsTests.cs @@ -0,0 +1,79 @@ +using Shouldly; +using Xunit; + +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests; + +public sealed class QuestDbHistorianOptionsTests +{ + private static QuestDbHistorianOptions Valid() => new() { ConnectionString = "Host=h;Port=8812" }; + + [Fact] + public void Defaults_are_the_documented_ones() + { + var o = new QuestDbHistorianOptions(); + o.ConnectionString.ShouldBe(""); + o.TablePrefix.ShouldBe("hist_"); + o.RetentionDays.ShouldBeNull(); + o.CommandTimeout.ShouldBe(TimeSpan.FromSeconds(30)); + o.AutoCreateSchema.ShouldBeTrue(); + o.VisibilityTimeout.ShouldBe(TimeSpan.FromSeconds(5)); + } + + [Fact] + public void A_valid_configuration_has_no_errors() => Valid().Validate().ShouldBeEmpty(); + + [Fact] + public void Missing_connection_string_names_the_key() + { + var o = Valid(); o.ConnectionString = " "; + o.Validate().ShouldHaveSingleItem().ShouldContain("ServerHistorian:QuestDb:ConnectionString"); + } + + [Theory] + [InlineData("")] + [InlineData("hist-")] + [InlineData("hist.")] + [InlineData("hist_; DROP")] + public void Bad_prefix_names_the_key(string prefix) + { + var o = Valid(); o.TablePrefix = prefix; + o.Validate().ShouldHaveSingleItem().ShouldContain("ServerHistorian:QuestDb:TablePrefix"); + } + + [Theory] + [InlineData("t0123456789ab_")] + [InlineData("A_9")] + public void Good_prefixes_pass(string prefix) + { + var o = Valid(); o.TablePrefix = prefix; + o.Validate().ShouldBeEmpty(); + } + + [Theory] + [InlineData(0)] + [InlineData(-3)] + public void Retention_below_one_names_the_key(int days) + { + var o = Valid(); o.RetentionDays = days; + o.Validate().ShouldHaveSingleItem().ShouldContain("ServerHistorian:QuestDb:RetentionDays"); + } + + [Fact] + public void Retention_of_one_is_valid() { var o = Valid(); o.RetentionDays = 1; o.Validate().ShouldBeEmpty(); } + + [Fact] + public void Non_positive_command_timeout_names_the_key() + { + var o = Valid(); o.CommandTimeout = TimeSpan.Zero; + o.Validate().ShouldHaveSingleItem().ShouldContain("ServerHistorian:QuestDb:CommandTimeout"); + } + + [Fact] + public void Negative_visibility_timeout_names_the_key_and_zero_is_valid() + { + var o = Valid(); o.VisibilityTimeout = TimeSpan.FromMilliseconds(-1); + o.Validate().ShouldHaveSingleItem().ShouldContain("ServerHistorian:QuestDb:VisibilityTimeout"); + o.VisibilityTimeout = TimeSpan.Zero; + o.Validate().ShouldBeEmpty(); + } +} diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests/QuestDbSchemaBootstrapperFailureTests.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests/QuestDbSchemaBootstrapperFailureTests.cs new file mode 100644 index 00000000..d1730696 --- /dev/null +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests/QuestDbSchemaBootstrapperFailureTests.cs @@ -0,0 +1,32 @@ +using Shouldly; +using Xunit; +using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common; + +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests; + +public sealed class QuestDbSchemaBootstrapperFailureTests +{ + private sealed class ManualClock : TimeProvider + { + private DateTimeOffset _now = new(2026, 10, 11, 0, 0, 0, TimeSpan.Zero); + public override DateTimeOffset GetUtcNow() => _now; + public void Advance(TimeSpan by) => _now += by; + } + + [Fact] + public async Task A_failure_is_cached_for_five_seconds_then_retried() + { + var opts = new QuestDbHistorianOptions { ConnectionString = "Host=127.0.0.1;Port=1;Timeout=1" }; + var clock = new ManualClock(); + var boot = new QuestDbSchemaBootstrapper(new QuestDbConnection(opts), opts, timeProvider: clock); + + var first = await Should.ThrowAsync(() => boot.EnsureAsync(CancellationToken.None)); + // Within the window the SAME failure is re-thrown without another connection attempt. + var second = await Should.ThrowAsync(() => boot.EnsureAsync(CancellationToken.None)); + second.ShouldBeSameAs(first); + + clock.Advance(BootstrapFailureCache.DefaultDuration + TimeSpan.FromSeconds(1)); + var third = await Should.ThrowAsync(() => boot.EnsureAsync(CancellationToken.None)); + third.ShouldNotBeSameAs(first); // the window closed: QuestDB was tried again + } +} diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests/QuestDbTablesTests.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests/QuestDbTablesTests.cs new file mode 100644 index 00000000..ef42fb50 --- /dev/null +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests/QuestDbTablesTests.cs @@ -0,0 +1,66 @@ +using Shouldly; +using Xunit; + +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests; + +public sealed class QuestDbTablesTests +{ + [Fact] + public void Sample_ddl_without_ttl_is_pinned() => + QuestDbTables.CreateSample("hist_", null).ShouldBe( + "CREATE TABLE IF NOT EXISTS hist_sample (ts TIMESTAMP, tag SYMBOL, kind BYTE, num_value DOUBLE, " + + "int_value LONG, str_value VARCHAR, status LONG, deleted BOOLEAN, server_ts TIMESTAMP) " + + "TIMESTAMP(ts) PARTITION BY DAY WAL DEDUP UPSERT KEYS(ts, tag)"); + + [Fact] + public void Sample_ddl_with_ttl_puts_ttl_between_partition_and_wal() => + QuestDbTables.CreateSample("hist_", 7).ShouldBe( + "CREATE TABLE IF NOT EXISTS hist_sample (ts TIMESTAMP, tag SYMBOL, kind BYTE, num_value DOUBLE, " + + "int_value LONG, str_value VARCHAR, status LONG, deleted BOOLEAN, server_ts TIMESTAMP) " + + "TIMESTAMP(ts) PARTITION BY DAY TTL 7 DAYS WAL DEDUP UPSERT KEYS(ts, tag)"); + + [Fact] + public void Event_ddl_is_pinned() + { + QuestDbTables.CreateEvent("hist_", null).ShouldBe( + "CREATE TABLE IF NOT EXISTS hist_event (ts TIMESTAMP, event_id VARCHAR, source SYMBOL, alarm_id VARCHAR, " + + "alarm_name VARCHAR, alarm_type_name VARCHAR, severity INT, event_kind VARCHAR, message VARCHAR, " + + "user_name VARCHAR, comment VARCHAR, received_utc TIMESTAMP) " + + "TIMESTAMP(ts) PARTITION BY DAY WAL DEDUP UPSERT KEYS(ts, event_id)"); + QuestDbTables.CreateEvent("hist_", 30).ShouldContain("PARTITION BY DAY TTL 30 DAYS WAL DEDUP"); + } + + [Fact] + public void Tag_ddl_is_pinned_and_never_carries_a_ttl() => + QuestDbTables.CreateTag("hist_").ShouldBe( + "CREATE TABLE IF NOT EXISTS hist_tag (ts TIMESTAMP, name SYMBOL, data_type BYTE, eng_unit VARCHAR, " + + "description VARCHAR, updated_utc TIMESTAMP) TIMESTAMP(ts) PARTITION BY YEAR WAL DEDUP UPSERT KEYS(ts, name)"); + + [Fact] + public void The_prefix_is_applied_to_every_table_name() + { + QuestDbTables.All("t1_").ShouldBe(["t1_sample", "t1_event", "t1_tag"]); + QuestDbTables.CreateAll("t1_", 3).ShouldAllBe(s => s.Contains("t1_")); + } + + [Fact] + public void An_invalid_prefix_is_refused() => + Should.Throw(() => QuestDbTables.CreateSample("a b", null)); + + [Fact] + public void Alter_ttl_is_pinned_and_null_removes_it() + { + QuestDbTables.AlterTtl("hist_sample", 30).ShouldBe("ALTER TABLE hist_sample SET TTL 30 DAYS"); + QuestDbTables.AlterTtl("hist_sample", null).ShouldBe("ALTER TABLE hist_sample SET TTL 0 DAYS"); + } + + [Theory] + [InlineData(1, "WEEK", 168)] // 7 days reads back as 1 WEEK + [InlineData(2, "WEEK", 336)] + [InlineData(30, "DAY", 720)] + [InlineData(0, "HOUR", 0)] // no TTL + [InlineData(5, "hour", 5)] + [InlineData(1, "MONTH", -1)] // unknown unit never equals a wanted value + public void Ttl_is_compared_in_hours(long value, string unit, long hours) => + QuestDbTables.TtlHours(value, unit).ShouldBe(hours); +} diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests/QuestDbWalVisibilityTests.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests/QuestDbWalVisibilityTests.cs new file mode 100644 index 00000000..b4f71714 --- /dev/null +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests/QuestDbWalVisibilityTests.cs @@ -0,0 +1,33 @@ +using Shouldly; +using Xunit; + +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests; + +public sealed class QuestDbWalVisibilityTests +{ + [Fact] + public async Task Timeout_zero_skips_the_wait_without_touching_the_database() + { + // Port 1 refuses connections: any attempt to query would throw, so returning true proves nothing was opened. + var conn = new QuestDbConnection(new QuestDbHistorianOptions { ConnectionString = "Host=127.0.0.1;Port=1;Timeout=1" }); + var wal = new QuestDbWalVisibility(conn, TimeSpan.Zero); + wal.Disabled.ShouldBeTrue(); + (await wal.WaitAsync("hist_sample", CancellationToken.None)).ShouldBeTrue(); + } + + [Fact] + public async Task A_positive_timeout_does_touch_the_database_control() + { + var conn = new QuestDbConnection(new QuestDbHistorianOptions { ConnectionString = "Host=127.0.0.1;Port=1;Timeout=1" }); + var wal = new QuestDbWalVisibility(conn, TimeSpan.FromSeconds(1)); + wal.Disabled.ShouldBeFalse(); + await Should.ThrowAsync(() => wal.WaitAsync("hist_sample", CancellationToken.None)); + } + + [Theory] + [InlineData(5, 5, true)] + [InlineData(6, 5, true)] + [InlineData(4, 5, false)] + public void Reached_is_writerTxn_at_or_beyond_the_target(long writer, long target, bool expected) => + QuestDbWalVisibility.Reached(writer, target).ShouldBe(expected); +} diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests.csproj b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests.csproj new file mode 100644 index 00000000..98c12258 --- /dev/null +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests.csproj @@ -0,0 +1,27 @@ + + + + net10.0 + enable + enable + false + true + ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests + + + + + + + + all + runtime; build; native; contentfiles; analyzers; buildtransitive + + + + + + + + + From ee174c0179b669ab8a63c9813343dc6699c40563 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Sun, 11 Oct 2026 02:04:48 -0400 Subject: [PATCH 11/23] feat(historian): QuestDB HistoryUpdate with soft delete and read-your-writes Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe --- .../QuestDbHistorianDataUpdater.cs | 258 ++++++++++++++++++ .../TombstoneTests.cs | 50 ++++ .../UpdateTests.cs | 235 ++++++++++++++++ 3 files changed, 543 insertions(+) create mode 100644 src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianDataUpdater.cs create mode 100644 tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/TombstoneTests.cs create mode 100644 tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/UpdateTests.cs diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianDataUpdater.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianDataUpdater.cs new file mode 100644 index 00000000..c818e8b0 --- /dev/null +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianDataUpdater.cs @@ -0,0 +1,258 @@ +using Microsoft.Extensions.Logging; +using Microsoft.Extensions.Logging.Abstractions; +using Npgsql; +using NpgsqlTypes; +using ZB.MOM.WW.OtOpcUa.Core.Abstractions; +using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common; + +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb; + +/// +/// OPC UA HistoryUpdate against the QuestDB sample table (spec §2.3). Every operation is an INSERT: the +/// table's DEDUP UPSERT KEYS(ts, tag) performs the replace, and a delete writes a tombstone +/// (deleted = true, NULL values, BadNoData). QuestDB UPDATE/DELETE are never issued. +/// Every status comes from , so codes equal the SQL Server backend's. +/// "Exists" means present and not deleted, so Insert over a tombstone succeeds (GoodEntryInserted) while +/// Replace and DeleteAtTime treat a tombstone as no entry. Existence is read with one range query and filtered +/// client-side (never an IN list). +/// Pre-1970 (RC1): QuestDB cannot store such a designated timestamp. Insert/Replace/Update of such an +/// item reports BadOutOfRange and sends nothing (a null/MinValue timestamp is still +/// BadInvalidTimestamp); DeleteAtTime reports BadNoEntryExists; a DeleteRaw window may start before +/// 1970 (read bounds are not clamped). +/// Read-your-writes: after writing, waits for the rows to become +/// visible, bounded by VisibilityTimeout; a timeout only logs a Warning, never changes a code. +/// +internal sealed class QuestDbHistorianDataUpdater : IHistorianDataUpdater +{ + /// Rows per INSERT and per DeleteRaw page (T0 Q2: 9,000 parameters, far under the 32,767 limit). + internal const int BatchSize = 1000; + + private const long ResolutionTicks = 10; + private readonly QuestDbConnection _conn; + private readonly QuestDbTagCatalog _tags; + private readonly QuestDbWalVisibility _wal; + private readonly ILogger _log; + + /// Creates the updater. + /// Connection helper. + /// Tag catalog (R7: the only known-tag authority). + /// WAL visibility wait (R5). + /// Optional logger. + public QuestDbHistorianDataUpdater(QuestDbConnection conn, QuestDbTagCatalog tags, QuestDbWalVisibility wal, + ILogger? log = null) + { + _conn = conn; + _tags = tags; + _wal = wal; + _log = log ?? NullLogger.Instance; + } + + /// + public async Task> UpdateDataAsync(string tag, HistoryUpdateMode mode, + IReadOnlyList values, CancellationToken ct) + { + ArgumentNullException.ThrowIfNull(tag); + ArgumentNullException.ThrowIfNull(values); + + // RC1: a valid-timestamp item before 1970 can never be stored; refuse it and plan the rest. + var final = new uint[values.Count]; + var kept = new List(values.Count); + for (var i = 0; i < values.Count; i++) + { + if (values[i].TimestampUtc is { } ts && ts != DateTime.MinValue && !QuestDbConnection.IsStorable(ts)) + final[i] = HistorianStatusCodes.BadOutOfRange; + else + kept.Add(i); + } + + var plan = HistoryUpdatePlanner.Plan(tag, mode, kept.Select(i => values[i]).ToList(), ResolutionTicks); + IReadOnlyList planned; + if (plan.Winners.Count == 0) + { + planned = HistoryUpdatePlanner.Complete(plan, mode, new Dictionary()); + } + else + { + var known = mode == HistoryUpdateMode.Replace + ? await _tags.ExistsAsync(tag, ct).ConfigureAwait(false) + : await EnsureAsync(tag, plan.Winners[0].Row.Value.Kind, ct).ConfigureAwait(false); + var outcome = known + ? await ApplyAsync(tag, mode, plan.Winners.Select(w => w.Row).ToList(), ct).ConfigureAwait(false) + : new Dictionary(); + planned = HistoryUpdatePlanner.Complete(plan, mode, outcome); + } + for (var k = 0; k < kept.Count; k++) final[kept[k]] = planned[k]; + return final; + } + + /// + public async Task DeleteRawAsync(string tag, DateTime startUtc, DateTime endUtc, CancellationToken ct) + { + ArgumentNullException.ThrowIfNull(tag); + if (!HistorianTagNames.IsValid(tag)) return HistorianStatusCodes.BadInvalidArgument; + if (!await _tags.ExistsAsync(tag, ct).ConfigureAwait(false)) + return HistoryUpdatePlanner.DeleteRawResult(tagKnown: false, 0); + + var cursor = QuestDbConnection.Round(startUtc); + var end = QuestDbConnection.Round(endUtc); + long total = 0; + while (cursor < end) + { + var page = new List(BatchSize); + await using (var conn = await _conn.OpenAsync(ct).ConfigureAwait(false)) + await using (var cmd = _conn.CreateCommand(conn, + $"SELECT ts FROM {_conn.SampleTable} WHERE tag = $1 AND ts >= $2 AND ts < $3 AND NOT deleted ORDER BY ts LIMIT {BatchSize}")) + { + QuestDbConnection.BindText(cmd, tag); + QuestDbConnection.BindTimestamp(cmd, cursor); + QuestDbConnection.BindTimestamp(cmd, end); + await using var rd = await cmd.ExecuteReaderAsync(ct).ConfigureAwait(false); + while (await rd.ReadAsync(ct).ConfigureAwait(false)) page.Add(QuestDbConnection.ReadUtc(rd, 0)); + } + if (page.Count == 0) break; + await WriteTombstonesAsync(tag, page, ct).ConfigureAwait(false); + // The wait between pages is required: without it the next page could re-read rows just tombstoned. + await _wal.WaitAsync(_conn.SampleTable, ct).ConfigureAwait(false); + total += page.Count; + if (page.Count < BatchSize) break; + cursor = page[^1].AddTicks(ResolutionTicks); + } + return HistoryUpdatePlanner.DeleteRawResult(tagKnown: true, total); + } + + /// + public async Task> DeleteAtTimeAsync(string tag, IReadOnlyList timesUtc, CancellationToken ct) + { + ArgumentNullException.ThrowIfNull(tag); + ArgumentNullException.ThrowIfNull(timesUtc); + if (!HistorianTagNames.IsValid(tag)) + { + var refused = new uint[timesUtc.Count]; + Array.Fill(refused, HistorianStatusCodes.BadInvalidArgument); + return refused; + } + + // M-2: round BEFORE both the delete and the comparison, so a sub-µs request time matches the row it deleted. + // MinValue stays as is: the planner turns it into BadInvalidTimestamp. + var rounded = timesUtc.Select(t => t == DateTime.MinValue ? t : QuestDbConnection.Round(t)).ToList(); + var candidates = new HashSet(); + foreach (var t in rounded) + if (t != DateTime.MinValue && t >= DateTime.UnixEpoch) candidates.Add(t.Ticks); // pre-1970 can hold no row + var deleted = new HashSet(); + if (candidates.Count == 0 || !await _tags.ExistsAsync(tag, ct).ConfigureAwait(false)) + return HistoryUpdatePlanner.DeleteAtTimeResults(rounded, deleted, tagKnown: false); + + var live = await ReadLiveAsync(tag, candidates.Min(), candidates.Max(), ct).ConfigureAwait(false); + var toDelete = candidates.Where(live.Contains).OrderBy(x => x).ToList(); + foreach (var chunk in toDelete.Chunk(BatchSize)) + await WriteTombstonesAsync(tag, chunk.Select(t => new DateTime(t, DateTimeKind.Utc)).ToList(), ct).ConfigureAwait(false); + if (toDelete.Count > 0) await _wal.WaitAsync(_conn.SampleTable, ct).ConfigureAwait(false); + deleted.UnionWith(toDelete); + return HistoryUpdatePlanner.DeleteAtTimeResults(rounded, deleted, tagKnown: true); + } + + private async Task EnsureAsync(string tag, HistorianValueKind kind, CancellationToken ct) + { + await _tags.EnsureAsync(tag, HistorianDataTypes.FromKind(kind), ct).ConfigureAwait(false); + return true; + } + + private async Task> ApplyAsync(string tag, HistoryUpdateMode mode, + List rows, CancellationToken ct) + { + // Key = the planner's tick (what Complete looks up); the stored time is Round(of the same row). + var live = await ReadLiveAsync(tag, rows.Min(r => QuestDbConnection.Round(r.TimeUtc).Ticks), + rows.Max(r => QuestDbConnection.Round(r.TimeUtc).Ticks), ct).ConfigureAwait(false); + var outcome = new Dictionary(); + var toWrite = new List(); + foreach (var r in rows) + { + var present = live.Contains(QuestDbConnection.Round(r.TimeUtc).Ticks); + if (mode == HistoryUpdateMode.Insert && present) continue; + if (mode == HistoryUpdateMode.Replace && !present) continue; + outcome[r.TimeUtc.Ticks] = present ? UpsertOutcome.Replaced : UpsertOutcome.Inserted; + toWrite.Add(r); + } + foreach (var chunk in toWrite.Chunk(BatchSize)) + await WriteRowsAsync(tag, chunk, ct).ConfigureAwait(false); + if (toWrite.Count > 0) await _wal.WaitAsync(_conn.SampleTable, ct).ConfigureAwait(false); + return outcome; + } + + /// Ticks of the live (present, not deleted) rows with ts in [minTicks, maxTicks]: one range query. + private async Task> ReadLiveAsync(string tag, long minTicks, long maxTicks, CancellationToken ct) + { + var live = new HashSet(); + await using var conn = await _conn.OpenAsync(ct).ConfigureAwait(false); + await using var cmd = _conn.CreateCommand(conn, + $"SELECT ts FROM {_conn.SampleTable} WHERE tag = $1 AND ts >= $2 AND ts <= $3 AND NOT deleted"); + QuestDbConnection.BindText(cmd, tag); + QuestDbConnection.BindTimestamp(cmd, new DateTime(minTicks, DateTimeKind.Utc)); + QuestDbConnection.BindTimestamp(cmd, new DateTime(maxTicks, DateTimeKind.Utc)); + await using var rd = await cmd.ExecuteReaderAsync(ct).ConfigureAwait(false); + while (await rd.ReadAsync(ct).ConfigureAwait(false)) live.Add(QuestDbConnection.ReadUtc(rd, 0).Ticks); + return live; + } + + private Task WriteRowsAsync(string tag, IReadOnlyList rows, CancellationToken ct) => + InsertAsync(rows.Count, ct, (cmd, i) => + { + var row = rows[i]; + var c = SampleColumnMapper.ToColumns(row.Value); + QuestDbConnection.BindTimestamp(cmd, row.TimeUtc, clampToEpoch: true); + QuestDbConnection.BindText(cmd, tag); + QuestDbConnection.BindByte(cmd, c.ValueKind); + Typed(cmd, NpgsqlDbType.Double, c.NumValue); + Typed(cmd, NpgsqlDbType.Bigint, c.IntValue); + Typed(cmd, NpgsqlDbType.Varchar, c.StrValue); + QuestDbConnection.BindLong(cmd, SampleColumnMapper.ToLongStatus(row.StatusCode)); + Typed(cmd, NpgsqlDbType.Boolean, false); + }); + + private Task WriteTombstonesAsync(string tag, IReadOnlyList times, CancellationToken ct) => + InsertAsync(times.Count, ct, (cmd, i) => + { + QuestDbConnection.BindTimestamp(cmd, times[i], clampToEpoch: true); + QuestDbConnection.BindText(cmd, tag); + QuestDbConnection.BindByte(cmd, (byte)HistorianValueKind.Empty); + Typed(cmd, NpgsqlDbType.Double, null); + Typed(cmd, NpgsqlDbType.Bigint, null); + Typed(cmd, NpgsqlDbType.Varchar, null); + QuestDbConnection.BindLong(cmd, SampleColumnMapper.ToLongStatus(HistorianStatusCodes.BadNoData)); + Typed(cmd, NpgsqlDbType.Boolean, true); + }); + + /// Multi-row INSERT; adds the first eight parameters of a row, this adds server_ts. + private async Task InsertAsync(int count, CancellationToken ct, Action bindRow) + { + if (count == 0) return; + await using var conn = await _conn.OpenAsync(ct).ConfigureAwait(false); + await using var cmd = _conn.CreateCommand(conn, ""); + var now = DateTime.UtcNow; + var sb = new System.Text.StringBuilder( + $"INSERT INTO {_conn.SampleTable} (ts, tag, kind, num_value, int_value, str_value, status, deleted, server_ts) VALUES "); + for (var i = 0; i < count; i++) + { + bindRow(cmd, i); + QuestDbConnection.BindTimestamp(cmd, now); + var b = i * 9; + if (i > 0) sb.Append(','); + sb.Append('('); + for (var p = 1; p <= 9; p++) sb.Append(p > 1 ? "," : "").Append('$').Append(b + p); + sb.Append(')'); + } + cmd.CommandText = sb.ToString(); + try + { + await cmd.ExecuteNonQueryAsync(ct).ConfigureAwait(false); + } + catch (Exception ex) when (ex is not OperationCanceledException) + { + _log.LogWarning(ex, "QuestDB HistoryUpdate INSERT of {Count} rows into {Table} failed.", count, _conn.SampleTable); + throw; + } + } + + private static void Typed(NpgsqlCommand cmd, NpgsqlDbType type, object? value) => + cmd.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = type, Value = value ?? DBNull.Value }); +} diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/TombstoneTests.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/TombstoneTests.cs new file mode 100644 index 00000000..2f3f53e3 --- /dev/null +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/TombstoneTests.cs @@ -0,0 +1,50 @@ +using Shouldly; +using Xunit; +using ZB.MOM.WW.OtOpcUa.Core.Abstractions; +using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common; + +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests; + +/// Soft-delete behaviour. The writer-redelivery case lives in T7 (TombstoneTests.Redelivery.cs). +[Collection(QuestDbCollection.Name)] +public sealed class TombstoneTests(QuestDbFixture fx) +{ + private static readonly DateTime T0 = new(2026, 3, 1, 12, 0, 0, DateTimeKind.Utc); + + [Fact] + public async Task Delete_then_raw_read_hides_the_row_and_leaves_a_NULL_valued_tombstone() + { + fx.SkipIfUnavailable(); + var rig = UpdateTests.MakeRig(fx); + await rig.Updater.UpdateDataAsync("t", HistoryUpdateMode.Insert, + [UpdateTests.V(T0, 1), UpdateTests.V(T0.AddSeconds(1), 2)], default); + (await rig.Updater.DeleteAtTimeAsync("t", [T0], default)).ShouldBe([HistorianStatusCodes.Good]); + + (await UpdateTests.RowsAsync(rig, "t", liveOnly: true)).Select(r => r.Num).ShouldBe([2d]); + var all = await UpdateTests.RowsAsync(rig, "t"); + all.Count.ShouldBe(2); // the tombstone replaced the row in place (dedup upsert) + var tomb = all[0]; + tomb.Deleted.ShouldBeTrue(); + tomb.Num.ShouldBeNull(); + tomb.Kind.ShouldBe((byte)0); + tomb.Status.ShouldBe((long)HistorianStatusCodes.BadNoData); + } + + [Fact] + public async Task Insert_over_a_tombstone_reports_GoodEntryInserted_and_the_row_is_visible() + { + fx.SkipIfUnavailable(); + var rig = UpdateTests.MakeRig(fx); + await rig.Updater.UpdateDataAsync("t", HistoryUpdateMode.Insert, [UpdateTests.V(T0, 1)], default); + await rig.Updater.DeleteAtTimeAsync("t", [T0], default); + + (await rig.Updater.UpdateDataAsync("t", HistoryUpdateMode.Replace, [UpdateTests.V(T0, 5)], default)) + .ShouldBe([HistorianStatusCodes.BadNoEntryExists]); // a tombstone is no entry for Replace + (await rig.Updater.UpdateDataAsync("t", HistoryUpdateMode.Insert, [UpdateTests.V(T0, 9)], default)) + .ShouldBe([HistorianStatusCodes.GoodEntryInserted]); + var live = await UpdateTests.RowsAsync(rig, "t", liveOnly: true); + live.Count.ShouldBe(1); + live[0].Num.ShouldBe(9d); + live[0].Deleted.ShouldBeFalse(); + } +} diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/UpdateTests.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/UpdateTests.cs new file mode 100644 index 00000000..c77f3d64 --- /dev/null +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/UpdateTests.cs @@ -0,0 +1,235 @@ +using Shouldly; +using Xunit; +using ZB.MOM.WW.OtOpcUa.Core.Abstractions; +using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common; + +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests; + +/// QuestDB-specific checks of (the literal codes run through conformance in T7). +[Collection(QuestDbCollection.Name)] +public sealed class UpdateTests(QuestDbFixture fx) +{ + private static readonly DateTime T0 = new(2026, 3, 1, 12, 0, 0, DateTimeKind.Utc); // µs-aligned + private const uint Good = HistorianStatusCodes.Good; + + internal sealed record Rig(QuestDbConnection Conn, QuestDbHistorianDataUpdater Updater, QuestDbTagCatalog Tags); + + internal static Rig MakeRig(QuestDbFixture fx, TimeSpan? visibility = null) + { + var o = fx.Options(fx.UniquePrefix()); + o.VisibilityTimeout = visibility ?? o.VisibilityTimeout; + var c = new QuestDbConnection(o); + var wal = new QuestDbWalVisibility(c, o.VisibilityTimeout); + var tags = new QuestDbTagCatalog(c, new QuestDbSchemaBootstrapper(c, o), wal); + return new Rig(c, new QuestDbHistorianDataUpdater(c, tags, wal), tags); + } + + internal static HistorizationValue V(DateTime? t, double v, uint status = Good) => + new(t, HistorianValue.FromDouble(v), status); + + internal sealed record Row(DateTime Ts, byte Kind, double? Num, long? Status, bool Deleted); + + /// Every stored row for the tag (tombstones included), straight from the table. + internal static async Task> RowsAsync(Rig rig, string tag, bool liveOnly = false) + { + await using var conn = await rig.Conn.OpenAsync(default); + await using var cmd = rig.Conn.CreateCommand(conn, + $"SELECT ts, kind, num_value, status, deleted FROM {rig.Conn.SampleTable} WHERE tag = $1 {(liveOnly ? "AND NOT deleted" : "")} ORDER BY ts"); + QuestDbConnection.BindText(cmd, tag); + var rows = new List(); + await using var r = await cmd.ExecuteReaderAsync(); + while (await r.ReadAsync()) + rows.Add(new Row(QuestDbConnection.ReadUtc(r, 0), QuestDbConnection.ReadKind(r, 1), + r.IsDBNull(2) ? null : r.GetDouble(2), r.IsDBNull(3) ? null : r.GetInt64(3), r.GetBoolean(4))); + return rows; + } + + internal static async Task TagRowsAsync(Rig rig, string tag) + { + await using var conn = await rig.Conn.OpenAsync(default); + await using var cmd = rig.Conn.CreateCommand(conn, $"SELECT count() FROM {rig.Conn.TagTable} WHERE name = $1"); + QuestDbConnection.BindText(cmd, tag); + return Convert.ToInt64(await cmd.ExecuteScalarAsync()); + } + + [Fact] + public async Task Insert_writes_absent_and_refuses_existing_and_is_visible_immediately() + { + fx.SkipIfUnavailable(); + var rig = MakeRig(fx); + (await rig.Updater.UpdateDataAsync("t1", HistoryUpdateMode.Insert, [V(T0, 1)], default)) + .ShouldBe([HistorianStatusCodes.GoodEntryInserted]); + var codes = await rig.Updater.UpdateDataAsync("t1", HistoryUpdateMode.Insert, + [V(T0, 99), V(T0.AddSeconds(1), 2)], default); + codes.ShouldBe([HistorianStatusCodes.BadEntryExists, HistorianStatusCodes.GoodEntryInserted]); + var rows = await RowsAsync(rig, "t1"); // read right after return: read-your-writes + rows.Select(r => r.Num).ShouldBe([1d, 2d]); + (await TagRowsAsync(rig, "t1")).ShouldBe(1); + } + + [Fact] + public async Task Replace_overwrites_present_refuses_missing_and_never_creates_the_tag() + { + fx.SkipIfUnavailable(); + var rig = MakeRig(fx); + (await rig.Updater.UpdateDataAsync("nope", HistoryUpdateMode.Replace, [V(T0, 1)], default)) + .ShouldBe([HistorianStatusCodes.BadNoEntryExists]); + (await TagRowsAsync(rig, "nope")).ShouldBe(0); + (await RowsAsync(rig, "nope")).ShouldBeEmpty(); + + await rig.Updater.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T0, 1)], default); + var codes = await rig.Updater.UpdateDataAsync("t", HistoryUpdateMode.Replace, + [V(T0, 5), V(T0.AddSeconds(1), 6)], default); + codes.ShouldBe([HistorianStatusCodes.GoodEntryReplaced, HistorianStatusCodes.BadNoEntryExists]); + (await RowsAsync(rig, "t")).Select(r => r.Num).ShouldBe([5d]); + } + + [Fact] + public async Task Update_upserts_both_ways_and_last_duplicate_wins() + { + fx.SkipIfUnavailable(); + var rig = MakeRig(fx); + await rig.Updater.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T0, 1)], default); + var codes = await rig.Updater.UpdateDataAsync("t", HistoryUpdateMode.Update, + [V(T0, 2), V(T0.AddSeconds(1), 3), V(T0.AddSeconds(1).AddTicks(3), 4)], default); + codes.ShouldBe([HistorianStatusCodes.GoodEntryReplaced, HistorianStatusCodes.GoodEntryInserted, + HistorianStatusCodes.GoodEntryInserted]); + (await RowsAsync(rig, "t")).Select(r => r.Num).ShouldBe([2d, 4d]); + } + + [Fact] + public async Task Same_microsecond_duplicates_collapse_before_the_database_sees_them() + { + fx.SkipIfUnavailable(); + var rig = MakeRig(fx); + // offsets 0 and 3 ticks sit inside one 10-tick window => the same stored µs + var codes = await rig.Updater.UpdateDataAsync("t", HistoryUpdateMode.Insert, + [V(T0, 1), V(T0.AddTicks(3), 2)], default); + codes.ShouldBe([HistorianStatusCodes.GoodEntryInserted, HistorianStatusCodes.BadEntryExists]); + (await RowsAsync(rig, "t")).Select(r => r.Num).ShouldBe([1d]); + } + + [Fact] + public async Task Invalid_items_get_the_planners_codes_and_the_rest_proceed() + { + fx.SkipIfUnavailable(); + var rig = MakeRig(fx); + var codes = await rig.Updater.UpdateDataAsync("t", HistoryUpdateMode.Insert, + [ + V(null, 1), V(DateTime.MinValue, 1), V(T0, double.NaN), + new HistorizationValue(T0.AddSeconds(1), HistorianValue.Empty, Good), + V(T0.AddSeconds(2), 7), + ], default); + codes.ShouldBe([HistorianStatusCodes.BadInvalidTimestamp, HistorianStatusCodes.BadInvalidTimestamp, + HistorianStatusCodes.BadInvalidArgument, HistorianStatusCodes.BadInvalidArgument, + HistorianStatusCodes.GoodEntryInserted]); + (await RowsAsync(rig, "t")).Count.ShouldBe(1); + } + + [Fact] + public async Task Over_long_tag_is_refused_and_nothing_is_written() + { + fx.SkipIfUnavailable(); + var rig = MakeRig(fx); + var tag = new string('x', HistorianTagNames.MaxLength + 1); + (await rig.Updater.UpdateDataAsync(tag, HistoryUpdateMode.Insert, [V(T0, 1)], default)) + .ShouldBe([HistorianStatusCodes.BadInvalidArgument]); + (await rig.Updater.DeleteAtTimeAsync(tag, [T0], default)).ShouldBe([HistorianStatusCodes.BadInvalidArgument]); + (await rig.Updater.DeleteRawAsync(tag, T0, T0.AddHours(1), default)).ShouldBe(HistorianStatusCodes.BadInvalidArgument); + } + + [Fact] + public async Task Pre_1970_items_are_BadOutOfRange_and_nothing_is_sent() + { + fx.SkipIfUnavailable(); + var rig = MakeRig(fx); + var pre = new DateTime(1969, 12, 31, 23, 59, 59, DateTimeKind.Utc); + foreach (var mode in new[] { HistoryUpdateMode.Insert, HistoryUpdateMode.Replace, HistoryUpdateMode.Update }) + { + var codes = await rig.Updater.UpdateDataAsync("t", mode, [V(pre, 1), V(DateTime.MinValue, 1), V(T0, 2)], default); + codes[0].ShouldBe(HistorianStatusCodes.BadOutOfRange); + codes[1].ShouldBe(HistorianStatusCodes.BadInvalidTimestamp); + } + // Pre-1970 only: the whole batch is refused, no tag row is created, no INSERT would succeed anyway. + (await rig.Updater.UpdateDataAsync("only-pre", HistoryUpdateMode.Update, [V(pre, 1)], default)) + .ShouldBe([HistorianStatusCodes.BadOutOfRange]); + (await TagRowsAsync(rig, "only-pre")).ShouldBe(0); + (await rig.Updater.DeleteAtTimeAsync("t", [pre], default)).ShouldBe([HistorianStatusCodes.BadNoEntryExists]); + } + + [Fact] + public async Task DeleteAtTime_tombstones_live_rows_and_reports_per_time_codes() + { + fx.SkipIfUnavailable(); + var rig = MakeRig(fx); + await rig.Updater.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T0, 1), V(T0.AddSeconds(1), 2)], default); + var codes = await rig.Updater.DeleteAtTimeAsync("t", + [T0, T0.AddSeconds(5), T0, DateTime.MinValue], default); + codes.ShouldBe([Good, HistorianStatusCodes.BadNoEntryExists, HistorianStatusCodes.BadNoEntryExists, + HistorianStatusCodes.BadInvalidTimestamp]); + (await RowsAsync(rig, "t", liveOnly: true)).Select(r => r.Num).ShouldBe([2d]); + // a tombstone is not a live row: deleting it again finds nothing + (await rig.Updater.DeleteAtTimeAsync("t", [T0], default)).ShouldBe([HistorianStatusCodes.BadNoEntryExists]); + (await rig.Updater.DeleteAtTimeAsync("unknown", [T0], default)).ShouldBe([HistorianStatusCodes.BadNoEntryExists]); + } + + [Fact] + public async Task DeleteAtTime_with_a_sub_microsecond_request_time_still_reports_Good_for_the_row_it_deleted() + { + fx.SkipIfUnavailable(); + var rig = MakeRig(fx); + await rig.Updater.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T0, 1)], default); + // +7 ticks is inside the same µs; the row lives at T0. + (await rig.Updater.DeleteAtTimeAsync("t", [T0.AddTicks(7)], default)).ShouldBe([Good]); + (await RowsAsync(rig, "t", liveOnly: true)).ShouldBeEmpty(); + } + + [Fact] + public async Task DeleteRaw_is_half_open_pages_through_big_ranges_and_reports_BadNoData_when_empty() + { + fx.SkipIfUnavailable(); + var rig = MakeRig(fx); + (await rig.Updater.DeleteRawAsync("unknown", T0, T0.AddHours(1), default)).ShouldBe(HistorianStatusCodes.BadNoData); + + // 2,500 rows => 3 pages of at most 1,000; plus one row exactly at the (exclusive) end. + var values = Enumerable.Range(0, 2500).Select(i => V(T0.AddSeconds(i), i)).ToList(); + values.Add(V(T0.AddSeconds(2500), -1)); + (await rig.Updater.UpdateDataAsync("big", HistoryUpdateMode.Insert, values, default)) + .ShouldAllBe(c => c == HistorianStatusCodes.GoodEntryInserted); + (await rig.Updater.DeleteRawAsync("big", T0, T0.AddSeconds(2500), default)).ShouldBe(Good); + var live = await RowsAsync(rig, "big", liveOnly: true); + live.Count.ShouldBe(1); + live[0].Num.ShouldBe(-1d); + (await rig.Updater.DeleteRawAsync("big", T0, T0.AddSeconds(2500), default)).ShouldBe(HistorianStatusCodes.BadNoData); + } + + [Fact] + public async Task DeleteRaw_with_a_pre_1970_start_reads_from_the_bound_unclamped() + { + fx.SkipIfUnavailable(); + var rig = MakeRig(fx); + await rig.Updater.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T0, 1), V(T0.AddSeconds(1), 2)], default); + (await rig.Updater.DeleteRawAsync("t", new DateTime(1900, 1, 1, 0, 0, 0, DateTimeKind.Utc), T0.AddSeconds(1), default)) + .ShouldBe(Good); + (await RowsAsync(rig, "t", liveOnly: true)).Select(r => r.Num).ShouldBe([2d]); + // DateTime.MinValue as the start is nudged inside, not bound as -infinity + (await rig.Updater.DeleteRawAsync("t", DateTime.MinValue, DateTime.MaxValue, default)).ShouldBe(Good); + (await RowsAsync(rig, "t", liveOnly: true)).ShouldBeEmpty(); + } + + [Fact] + public async Task Read_after_write_is_visible_across_many_immediate_calls() + { + fx.SkipIfUnavailable(); + var rig = MakeRig(fx); + // ~15% of un-waited inserts are not yet visible (spike Q4), so 40 round trips expose a missing wait. + for (var i = 0; i < 40; i++) + { + var t = T0.AddSeconds(i); + (await rig.Updater.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(t, i)], default)) + .ShouldBe([HistorianStatusCodes.GoodEntryInserted]); + (await rig.Updater.UpdateDataAsync("t", HistoryUpdateMode.Replace, [V(t, i + 1000)], default)) + .ShouldBe([HistorianStatusCodes.GoodEntryReplaced], $"iteration {i}: the insert was not visible"); + } + } +} From 4c053cc4d0f9e5e2234e6412f07a70e923ab7dc3 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Sun, 11 Oct 2026 02:08:04 -0400 Subject: [PATCH 12/23] feat(historian): QuestDB live writes, provisioning and alarm events Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe --- .../QuestDbAlarmHistorianWriter.cs | 130 +++++++++++ .../QuestDbHistorianValueWriter.cs | 142 +++++++++++ .../QuestDbTagProvisioner.cs | 141 +++++++++++ .../AlarmWriterTests.cs | 118 ++++++++++ .../ProvisioningTests.cs | 102 ++++++++ .../WriteTests.cs | 221 ++++++++++++++++++ 6 files changed, 854 insertions(+) create mode 100644 src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbAlarmHistorianWriter.cs create mode 100644 src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianValueWriter.cs create mode 100644 src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbTagProvisioner.cs create mode 100644 tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/AlarmWriterTests.cs create mode 100644 tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/ProvisioningTests.cs create mode 100644 tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/WriteTests.cs diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbAlarmHistorianWriter.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbAlarmHistorianWriter.cs new file mode 100644 index 00000000..9b39a5d6 --- /dev/null +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbAlarmHistorianWriter.cs @@ -0,0 +1,130 @@ +using Microsoft.Extensions.Logging; +using Microsoft.Extensions.Logging.Abstractions; +using Npgsql; +using NpgsqlTypes; +using ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian; +using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common; + +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb; + +/// +/// over the QuestDB event table. Each event carries the deterministic +/// and is inserted through the table's DEDUP UPSERT KEYS(ts, event_id) +/// (T0 Q3 confirmed VARCHAR works as a key), so the store-and-forward drain's at-least-once redelivery, and both +/// pair nodes writing the same event, converge on one row (the redelivered row is identical except +/// received_utc). Outcomes: Ack on insert; RetryPlease on transient / unclassified errors and on +/// cancellation; PermanentFail only for data the server will always reject (a time before 1970, which is not a +/// storable designated timestamp, or a server message naming an inconvertible / out-of-range value). Never throws. +/// +internal sealed class QuestDbAlarmHistorianWriter : IAlarmHistorianWriter +{ + private static readonly string[] PermanentMessageFragments = + ["designated timestamp", "inconvertible", "out of range", "too long", "cannot be null"]; + + private readonly QuestDbConnection _conn; + private readonly QuestDbSchemaBootstrapper _bootstrap; + private readonly QuestDbWalVisibility _wal; + private readonly ILogger _log; + private readonly TimeProvider _clock; + + /// Creates the writer. + /// Connection helper. + /// Schema bootstrapper. + /// WAL visibility wait (once per batch, so a following read sees the events). + /// Optional logger. + /// Clock for received_utc. + public QuestDbAlarmHistorianWriter(QuestDbConnection conn, QuestDbSchemaBootstrapper bootstrap, QuestDbWalVisibility wal, + ILogger? log = null, TimeProvider? clock = null) + { + _conn = conn; + _bootstrap = bootstrap; + _wal = wal; + _log = log ?? NullLogger.Instance; + _clock = clock ?? TimeProvider.System; + } + + /// + public async Task> WriteBatchAsync(IReadOnlyList batch, CancellationToken cancellationToken) + { + ArgumentNullException.ThrowIfNull(batch); + if (batch.Count == 0) return []; + var outcomes = new HistorianWriteOutcome[batch.Count]; + try + { + await _bootstrap.EnsureAsync(cancellationToken).ConfigureAwait(false); + } + catch (Exception ex) + { + _log.LogWarning("QuestDB alarm historian bootstrap failed ({Exception}); will retry.", ex.GetType().Name); + Array.Fill(outcomes, HistorianWriteOutcome.RetryPlease); + return outcomes; + } + + var wrote = false; + for (var i = 0; i < batch.Count; i++) + { + if (cancellationToken.IsCancellationRequested) { outcomes[i] = HistorianWriteOutcome.RetryPlease; continue; } + outcomes[i] = await WriteOneAsync(batch[i], cancellationToken).ConfigureAwait(false); + wrote |= outcomes[i] == HistorianWriteOutcome.Ack; + } + + if (wrote) + { + try { await _wal.WaitAsync(_conn.EventTable, cancellationToken).ConfigureAwait(false); } + catch (Exception ex) { _log.LogDebug("QuestDB alarm visibility wait failed ({Exception}).", ex.GetType().Name); } + } + return outcomes; + } + + private async Task WriteOneAsync(AlarmHistorianEvent e, CancellationToken ct) + { + try + { + if (!QuestDbConnection.IsStorable(e.TimestampUtc)) + { + _log.LogWarning("QuestDB alarm event dropped: its time is before 1970-01-01 and cannot be stored."); + return HistorianWriteOutcome.PermanentFail; + } + + await using var conn = await _conn.OpenAsync(ct).ConfigureAwait(false); + await using var cmd = _conn.CreateCommand(conn, + $"INSERT INTO {_conn.EventTable} (ts, event_id, source, alarm_id, alarm_name, alarm_type_name, severity, event_kind, message, user_name, comment, received_utc) " + + "VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12)"); + QuestDbConnection.BindTimestamp(cmd, e.TimestampUtc); + QuestDbConnection.BindText(cmd, AlarmEventIds.ComputeEventId(e)); + QuestDbConnection.BindText(cmd, e.EquipmentPath ?? ""); + QuestDbConnection.BindText(cmd, e.AlarmId ?? ""); + QuestDbConnection.BindText(cmd, e.AlarmName ?? ""); + QuestDbConnection.BindText(cmd, e.AlarmTypeName ?? ""); + cmd.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Integer, Value = AlarmEventIds.ToNumericSeverity(e.Severity) }); + QuestDbConnection.BindText(cmd, e.EventKind ?? ""); + QuestDbConnection.BindText(cmd, e.Message); + QuestDbConnection.BindText(cmd, e.User); + QuestDbConnection.BindText(cmd, e.Comment); + QuestDbConnection.BindTimestamp(cmd, _clock.GetUtcNow().UtcDateTime); + await cmd.ExecuteNonQueryAsync(ct).ConfigureAwait(false); + return HistorianWriteOutcome.Ack; + } + catch (OperationCanceledException) when (ct.IsCancellationRequested) + { + return HistorianWriteOutcome.RetryPlease; + } + catch (Exception ex) + { + var outcome = Classify(ex); + _log.LogWarning("QuestDB alarm event write failed ({Exception}); outcome {Outcome}.", ex.GetType().Name, outcome); + return outcome; + } + } + + // Only a data error QuestDB will reject forever dead-letters; anything unknown or non-database retries (never lose alarm + // history). SQLSTATE is always 00000 on QuestDB, so a server rejection is classified by message text. + internal static HistorianWriteOutcome Classify(Exception ex) + { + if (QuestDbConnection.IsTransient(ex)) return HistorianWriteOutcome.RetryPlease; + if (ex is PostgresException pg && + PermanentMessageFragments.Any(f => pg.MessageText.Contains(f, StringComparison.OrdinalIgnoreCase))) + return HistorianWriteOutcome.PermanentFail; + return HistorianWriteOutcome.RetryPlease; + } +} diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianValueWriter.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianValueWriter.cs new file mode 100644 index 00000000..de1d790c --- /dev/null +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianValueWriter.cs @@ -0,0 +1,142 @@ +using System.Text; +using Microsoft.Extensions.Logging; +using Microsoft.Extensions.Logging.Abstractions; +using Npgsql; +using NpgsqlTypes; +using ZB.MOM.WW.OtOpcUa.Core.Abstractions; +using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common; + +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb; + +/// +/// over the QuestDB sample table. Redelivery-idempotent with first-wins +/// semantics: QuestDB's DEDUP UPSERT KEYS(ts, tag) would let a re-sent sample overwrite the stored one (and a +/// re-sent live sample resurrect a tombstone), so the writer reads which timestamps already exist for the tag in +/// the batch's range (tombstones count as present, T0 Q11 range form) and inserts only the absent rows. +/// Never throws; returns false on any failure so the recorder keeps its entries. +/// +/// +/// Batch normalisation is at µs resolution (non-finite doubles become Empty +/// + Bad, an in-batch µs collision keeps the first). RC1: a sample whose time is before 1970 cannot be a +/// QuestDB designated timestamp and one such row would fail the whole multi-row INSERT, so those samples are dropped +/// (Warning once per batch, counted in ) and the call still succeeds for the rest. +/// R5: after inserting, the writer waits for WAL visibility so an immediate redelivery sees the rows. +/// Inserts are chunked at rows (8 bound parameters per row, far under the 32,767 limit). +/// +internal sealed class QuestDbHistorianValueWriter : IHistorianValueWriter +{ + /// Rows per multi-row INSERT (T0 Q2: 1,000 rows is reliable). + internal const int BatchRows = 1000; + + private readonly QuestDbConnection _conn; + private readonly QuestDbTagCatalog _catalog; + private readonly QuestDbWalVisibility _wal; + private readonly ILogger _log; + private readonly Func _utcNow; + private long _droppedPre1970; + + /// Creates the writer. + /// Connection helper. + /// Tag catalog (the only tag authority). + /// WAL visibility wait. + /// Optional logger. + /// Clock for null timestamps and server_ts (tests). + public QuestDbHistorianValueWriter(QuestDbConnection conn, QuestDbTagCatalog catalog, QuestDbWalVisibility wal, + ILogger? log = null, Func? utcNow = null) + { + _conn = conn; + _catalog = catalog; + _wal = wal; + _log = log ?? NullLogger.Instance; + _utcNow = utcNow ?? (() => DateTime.UtcNow); + } + + /// Samples dropped so far because their time is before 1970 (RC1). + internal long DroppedPre1970 => Interlocked.Read(ref _droppedPre1970); + + /// + public async Task WriteLiveValuesAsync(string tag, IReadOnlyList values, CancellationToken ct) + { + try + { + if (!HistorianTagNames.IsValid(tag)) + { + _log.LogWarning("QuestDB historian live-value write refused: tag name is missing or longer than {Max} characters.", + HistorianTagNames.MaxLength); + return false; + } + if (values is null || values.Count == 0) return true; + + var prepared = LiveWritePreparer.Prepare(values, _utcNow, resolutionTicks: 10); + var rows = prepared.Where(r => QuestDbConnection.IsStorable(r.TimeUtc)).ToList(); + var dropped = prepared.Count - rows.Count; + if (dropped > 0) + { + Interlocked.Add(ref _droppedPre1970, dropped); + _log.LogWarning("QuestDB historian dropped {Count} sample(s) for a tag because their time is before 1970-01-01 (not storable).", dropped); + } + if (rows.Count == 0) return true; + + var dataType = HistorianDataTypes.FromKind(rows.Select(r => r.Value.Kind).FirstOrDefault(k => k != HistorianValueKind.Empty)); + await _catalog.EnsureAsync(tag, dataType, ct).ConfigureAwait(false); + + var present = await ExistingTicksAsync(tag, rows, ct).ConfigureAwait(false); + var fresh = rows.Where(r => !present.Contains(QuestDbConnection.Round(r.TimeUtc).Ticks)).ToList(); + if (fresh.Count == 0) return true; + + var serverTs = _utcNow(); + for (var i = 0; i < fresh.Count; i += BatchRows) + await InsertAsync(tag, fresh.GetRange(i, Math.Min(BatchRows, fresh.Count - i)), serverTs, ct).ConfigureAwait(false); + + await _wal.WaitAsync(_conn.SampleTable, ct).ConfigureAwait(false); // timeout/suspension does not change the result (R5) + return true; + } + catch (Exception ex) + { + _log.LogWarning("QuestDB historian live-value write failed ({Exception}); will retry.", ex.GetType().Name); + return false; + } + } + + // T0 Q11 range form: tombstones are NOT filtered out (a deleted row must not be resurrected). + private async Task> ExistingTicksAsync(string tag, List rows, CancellationToken ct) + { + var min = rows.Min(r => QuestDbConnection.Round(r.TimeUtc)); + var max = rows.Max(r => QuestDbConnection.Round(r.TimeUtc)); + var set = new HashSet(); + await using var conn = await _conn.OpenAsync(ct).ConfigureAwait(false); + await using var cmd = _conn.CreateCommand(conn, $"SELECT ts FROM {_conn.SampleTable} WHERE tag = $1 AND ts >= $2 AND ts <= $3"); + QuestDbConnection.BindText(cmd, tag); + QuestDbConnection.BindTimestamp(cmd, min); + QuestDbConnection.BindTimestamp(cmd, max); + await using var r = await cmd.ExecuteReaderAsync(ct).ConfigureAwait(false); + while (await r.ReadAsync(ct).ConfigureAwait(false)) set.Add(QuestDbConnection.ReadUtc(r, 0).Ticks); + return set; + } + + private async Task InsertAsync(string tag, List chunk, DateTime serverTs, CancellationToken ct) + { + var sql = new StringBuilder($"INSERT INTO {_conn.SampleTable} (ts, tag, kind, num_value, int_value, str_value, status, deleted, server_ts) VALUES "); + await using var conn = await _conn.OpenAsync(ct).ConfigureAwait(false); + await using var cmd = _conn.CreateCommand(conn, ""); + var n = 1; + for (var i = 0; i < chunk.Count; i++) + { + if (i > 0) sql.Append(','); + sql.Append($"(${n},${n + 1},${n + 2},${n + 3},${n + 4},${n + 5},${n + 6},false,${n + 7})"); + n += 8; + var row = chunk[i]; + var c = SampleColumnMapper.ToColumns(row.Value); + QuestDbConnection.BindTimestamp(cmd, row.TimeUtc); + QuestDbConnection.BindText(cmd, tag); + QuestDbConnection.BindByte(cmd, c.ValueKind); + cmd.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Double, Value = c.NumValue is { } d ? d : DBNull.Value }); + cmd.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Bigint, Value = c.IntValue is { } l ? l : DBNull.Value }); + QuestDbConnection.BindText(cmd, c.StrValue); + QuestDbConnection.BindLong(cmd, SampleColumnMapper.ToLongStatus(row.StatusCode)); + QuestDbConnection.BindTimestamp(cmd, serverTs); + } + cmd.CommandText = sql.ToString(); + await cmd.ExecuteNonQueryAsync(ct).ConfigureAwait(false); + } +} diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbTagProvisioner.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbTagProvisioner.cs new file mode 100644 index 00000000..f32f9828 --- /dev/null +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbTagProvisioner.cs @@ -0,0 +1,141 @@ +using System.Text; +using Microsoft.Extensions.Logging; +using Microsoft.Extensions.Logging.Abstractions; +using ZB.MOM.WW.OtOpcUa.Core.Abstractions; +using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common; + +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb; + +/// +/// for the QuestDB backend: upserts tag rows (fixed designated timestamp +/// 1970-01-01, so the dedup key makes it an upsert by name). QuestDB replaces the WHOLE row on a key hit, so a null +/// engineering unit / description in a request would wipe the stored one; per chunk the existing rows are read +/// (WHERE name IN (…)) and COALESCE(request, stored) is applied client-side before inserting. +/// Every data type is provisionable (Skipped is always 0). Never throws. After each chunk it waits for WAL +/// visibility and tells the catalog the names are known (R7). +/// +internal sealed class QuestDbTagProvisioner : IHistorianProvisioning +{ + private const int ChunkSize = 400; // 6 parameters per tag on insert, 400 on the IN list + private const int MaxUnitLength = 64; + private const int MaxDescriptionLength = 400; + + private readonly QuestDbConnection _conn; + private readonly QuestDbSchemaBootstrapper _bootstrap; + private readonly QuestDbTagCatalog _catalog; + private readonly QuestDbWalVisibility _wal; + private readonly ILogger _log; + private readonly TimeProvider _clock; + + /// Creates the provisioner. + /// Connection helper. + /// Schema bootstrapper. + /// Tag catalog (told which names are now known). + /// WAL visibility wait. + /// Optional logger. + /// Clock for updated_utc. + public QuestDbTagProvisioner(QuestDbConnection conn, QuestDbSchemaBootstrapper bootstrap, QuestDbTagCatalog catalog, + QuestDbWalVisibility wal, ILogger? log = null, TimeProvider? clock = null) + { + _conn = conn; + _bootstrap = bootstrap; + _catalog = catalog; + _wal = wal; + _log = log ?? NullLogger.Instance; + _clock = clock ?? TimeProvider.System; + } + + /// + public async Task EnsureTagsAsync(IReadOnlyList requests, CancellationToken ct) + { + var requested = requests?.Count ?? 0; + if (requested == 0) return new HistorianProvisionResult(0, 0, 0, 0); + var failed = 0; + var ensured = 0; + try + { + await _bootstrap.EnsureAsync(ct).ConfigureAwait(false); + + // Distinct names (last request wins); over-long/empty names fail alone. Weight = requests folded into the name. + var byName = new Dictionary(StringComparer.Ordinal); + foreach (var r in requests!) + { + if (r is null || string.IsNullOrWhiteSpace(r.TagName) || !HistorianTagNames.IsValid(r.TagName)) { failed++; continue; } + byName[r.TagName] = (r, byName.TryGetValue(r.TagName, out var e) ? e.Count + 1 : 1); + } + + var ordered = byName.OrderBy(kv => kv.Key, StringComparer.Ordinal).Select(kv => kv.Value).ToList(); + for (var i = 0; i < ordered.Count; i += ChunkSize) + { + var chunk = ordered.Skip(i).Take(ChunkSize).ToList(); + var weight = chunk.Sum(c => c.Count); + if (await TryUpsertAsync(chunk.Select(c => c.Req).ToList(), ct).ConfigureAwait(false)) ensured += weight; + else failed += weight; + } + } + catch (Exception ex) + { + _log.LogWarning("QuestDB historian provisioning failed ({Exception}).", ex.GetType().Name); + failed = requested - ensured; + } + + return new HistorianProvisionResult(requested, ensured, 0, failed); + } + + private async Task TryUpsertAsync(List chunk, CancellationToken ct) + { + try + { + var stored = new Dictionary(StringComparer.Ordinal); + await using (var conn = await _conn.OpenAsync(ct).ConfigureAwait(false)) + { + var sel = new StringBuilder($"SELECT name, eng_unit, description FROM {_conn.TagTable} WHERE name IN ("); + await using var cmd = _conn.CreateCommand(conn, ""); + for (var i = 0; i < chunk.Count; i++) + { + sel.Append(i == 0 ? "$" : ",$").Append(i + 1); + QuestDbConnection.BindText(cmd, chunk[i].TagName); + } + cmd.CommandText = sel.Append(')').ToString(); + await using var r = await cmd.ExecuteReaderAsync(ct).ConfigureAwait(false); + while (await r.ReadAsync(ct).ConfigureAwait(false)) + stored[r.GetString(0)] = (r.IsDBNull(1) ? null : r.GetString(1), r.IsDBNull(2) ? null : r.GetString(2)); + } + + var now = _clock.GetUtcNow().UtcDateTime; + await using (var conn = await _conn.OpenAsync(ct).ConfigureAwait(false)) + { + var ins = new StringBuilder($"INSERT INTO {_conn.TagTable} (ts, name, data_type, eng_unit, description, updated_utc) VALUES "); + await using var cmd = _conn.CreateCommand(conn, ""); + var n = 1; + for (var i = 0; i < chunk.Count; i++) + { + var req = chunk[i]; + stored.TryGetValue(req.TagName, out var old); + if (i > 0) ins.Append(','); + ins.Append($"(${n},${n + 1},${n + 2},${n + 3},${n + 4},${n + 5})"); + n += 6; + QuestDbConnection.BindTimestamp(cmd, QuestDbTables.TagEpoch); + QuestDbConnection.BindText(cmd, req.TagName); + QuestDbConnection.BindByte(cmd, (byte)req.DataType); + QuestDbConnection.BindText(cmd, Clip(req.EngineeringUnit ?? old.Unit, MaxUnitLength)); + QuestDbConnection.BindText(cmd, Clip(req.Description ?? old.Desc, MaxDescriptionLength)); + QuestDbConnection.BindTimestamp(cmd, now); + } + cmd.CommandText = ins.ToString(); + await cmd.ExecuteNonQueryAsync(ct).ConfigureAwait(false); + } + + await _wal.WaitAsync(_conn.TagTable, ct).ConfigureAwait(false); + foreach (var c in chunk) _catalog.MarkKnown(c.TagName); + return true; + } + catch (Exception ex) + { + _log.LogWarning("QuestDB historian tag upsert failed ({Exception}).", ex.GetType().Name); + return false; + } + } + + private static string? Clip(string? s, int max) => s is null ? null : s.Length <= max ? s : s[..max]; +} diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/AlarmWriterTests.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/AlarmWriterTests.cs new file mode 100644 index 00000000..032ffa86 --- /dev/null +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/AlarmWriterTests.cs @@ -0,0 +1,118 @@ +using Npgsql; +using Shouldly; +using Xunit; +using ZB.MOM.WW.OtOpcUa.Core.Abstractions; +using ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian; +using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common; + +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests; + +[Collection(QuestDbCollection.Name)] +public sealed class AlarmWriterTests(QuestDbFixture fx) +{ + // µs-aligned: QuestDB stores µs, so the ts column reads back exactly. + private static readonly DateTime T = new DateTime(2026, 10, 9, 1, 2, 3, DateTimeKind.Utc).AddTicks(1230); + + private (QuestDbAlarmHistorianWriter W, QuestDbConnection C) Make(string? connection = null, bool auto = true) + { + var o = fx.Options(fx.UniquePrefix(), autoCreate: auto); + if (connection is not null) o.ConnectionString = connection; + var c = new QuestDbConnection(o); + return (new QuestDbAlarmHistorianWriter(c, new QuestDbSchemaBootstrapper(c, o), new QuestDbWalVisibility(c, o.VisibilityTimeout)), c); + } + + private static AlarmHistorianEvent Ev(string kind = "Activated", string type = "LimitAlarm", string? comment = "c", DateTime? at = null) => + new("alarm-1", "Plant/Line/Eq", "HiTemp", type, AlarmSeverity.High, kind, "msg", "user", comment, at ?? T); + + private static async Task CountAsync(QuestDbConnection c, string? kind = null) + { + await using var conn = await c.OpenAsync(default); + await using var cmd = c.CreateCommand(conn, $"SELECT count() FROM {c.EventTable}" + (kind is null ? "" : " WHERE event_kind = $1")); + if (kind is not null) QuestDbConnection.BindText(cmd, kind); + return Convert.ToInt64(await cmd.ExecuteScalarAsync()); + } + + [Fact] + public async Task Event_round_trips_and_a_repeat_is_one_row_with_every_write_Ack() + { + fx.SkipIfUnavailable(); + var (w, c) = Make(); + (await w.WriteBatchAsync([Ev(), Ev()], CancellationToken.None)).ShouldBe([HistorianWriteOutcome.Ack, HistorianWriteOutcome.Ack]); + (await w.WriteBatchAsync([Ev()], CancellationToken.None)).ShouldBe([HistorianWriteOutcome.Ack]); + (await w.WriteBatchAsync([Ev(kind: "Cleared")], CancellationToken.None)).ShouldBe([HistorianWriteOutcome.Ack]); + + await using var conn = await c.OpenAsync(default); + await using var cmd = c.CreateCommand(conn, + $"SELECT event_id, source, alarm_id, alarm_name, alarm_type_name, severity, message, user_name, comment, ts, received_utc FROM {c.EventTable} WHERE event_kind = 'Activated'"); + await using var r = await cmd.ExecuteReaderAsync(); + (await r.ReadAsync()).ShouldBeTrue(); + r.GetString(0).ShouldBe(AlarmEventIds.ComputeEventId(Ev())); + r.GetString(1).ShouldBe("Plant/Line/Eq"); + r.GetString(2).ShouldBe("alarm-1"); + r.GetString(3).ShouldBe("HiTemp"); + r.GetString(4).ShouldBe("LimitAlarm"); + r.GetInt32(5).ShouldBe(AlarmEventIds.ToNumericSeverity(AlarmSeverity.High)); + r.GetInt32(5).ShouldBe(700); + r.GetString(6).ShouldBe("msg"); + r.GetString(7).ShouldBe("user"); + r.GetString(8).ShouldBe("c"); + QuestDbConnection.ReadUtc(r, 9).ShouldBe(T); + (DateTime.UtcNow - QuestDbConnection.ReadUtc(r, 10)).ShouldBeLessThan(TimeSpan.FromMinutes(1)); + (await r.ReadAsync()).ShouldBeFalse(); + await r.CloseAsync(); + (await CountAsync(c)).ShouldBe(2); + } + + [Fact] + public async Task A_null_comment_and_a_very_long_type_name_are_both_stored_and_Ack() + { + fx.SkipIfUnavailable(); + var (w, c) = Make(); + var r = await w.WriteBatchAsync([Ev(kind: "A"), Ev(kind: "B", type: new string('t', 500)), Ev(kind: "C", comment: null)], CancellationToken.None); + r.ShouldBe([HistorianWriteOutcome.Ack, HistorianWriteOutcome.Ack, HistorianWriteOutcome.Ack]); + (await CountAsync(c)).ShouldBe(3); + } + + [Fact] + public async Task An_event_before_1970_is_PermanentFail_and_siblings_still_Ack() + { + fx.SkipIfUnavailable(); + var (w, c) = Make(); + var r = await w.WriteBatchAsync([Ev(kind: "A"), Ev(kind: "B", at: new DateTime(1960, 1, 1, 0, 0, 0, DateTimeKind.Utc)), Ev(kind: "C")], CancellationToken.None); + r.ShouldBe([HistorianWriteOutcome.Ack, HistorianWriteOutcome.PermanentFail, HistorianWriteOutcome.Ack]); + (await CountAsync(c)).ShouldBe(2); + } + + [Fact] + public async Task Concurrent_writers_of_the_same_event_all_Ack_with_one_row() + { + fx.SkipIfUnavailable(); + var (w, c) = Make(); + await w.WriteBatchAsync([Ev(kind: "warm")], CancellationToken.None); + var rs = await Task.WhenAll(Enumerable.Range(0, 12).Select(_ => Task.Run(() => w.WriteBatchAsync([Ev()], CancellationToken.None)))); + rs.ShouldAllBe(x => x[0] == HistorianWriteOutcome.Ack); + (await CountAsync(c, "Activated")).ShouldBe(1); + } + + [Fact] + public async Task Unreachable_database_is_RetryPlease_not_an_exception_and_empty_batch_is_empty() + { + fx.SkipIfUnavailable(); + var (w, _) = Make("Host=localhost;Port=1;Username=admin;Password=quest;Database=qdb;ServerCompatibilityMode=NoTypeLoading;Timeout=2", auto: false); + (await w.WriteBatchAsync([Ev()], CancellationToken.None)).ShouldBe([HistorianWriteOutcome.RetryPlease]); + (await w.WriteBatchAsync([], CancellationToken.None)).ShouldBeEmpty(); + } + + [Fact] + public void Classify_maps_unknown_and_transport_errors_to_RetryPlease_and_data_rejections_to_PermanentFail() + { + QuestDbAlarmHistorianWriter.Classify(new InvalidOperationException("boom")).ShouldBe(HistorianWriteOutcome.RetryPlease); + QuestDbAlarmHistorianWriter.Classify(new IOException("reset")).ShouldBe(HistorianWriteOutcome.RetryPlease); + QuestDbAlarmHistorianWriter.Classify(new TimeoutException()).ShouldBe(HistorianWriteOutcome.RetryPlease); + QuestDbAlarmHistorianWriter.Classify(new PostgresException("table is busy", "ERROR", "ERROR", "00000")).ShouldBe(HistorianWriteOutcome.RetryPlease); + QuestDbAlarmHistorianWriter.Classify(new PostgresException("designated timestamp before 1970-01-01 is not allowed", "ERROR", "ERROR", "00000")) + .ShouldBe(HistorianWriteOutcome.PermanentFail); + QuestDbAlarmHistorianWriter.Classify(new PostgresException("inconvertible value: x [VARCHAR -> INT]", "ERROR", "ERROR", "00000")) + .ShouldBe(HistorianWriteOutcome.PermanentFail); + } +} diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/ProvisioningTests.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/ProvisioningTests.cs new file mode 100644 index 00000000..e1b8605e --- /dev/null +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/ProvisioningTests.cs @@ -0,0 +1,102 @@ +using Shouldly; +using Xunit; +using ZB.MOM.WW.OtOpcUa.Core.Abstractions; + +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests; + +[Collection(QuestDbCollection.Name)] +public sealed class ProvisioningTests(QuestDbFixture fx) +{ + private (QuestDbTagProvisioner P, QuestDbConnection C, QuestDbTagCatalog Cat) Make(string? prefix = null, string? connectionString = null, + bool autoCreate = true) + { + var o = fx.Options(prefix ?? fx.UniquePrefix(), autoCreate: autoCreate); + if (connectionString is not null) o.ConnectionString = connectionString; + var c = new QuestDbConnection(o); + var b = new QuestDbSchemaBootstrapper(c, o); + var wal = new QuestDbWalVisibility(c, o.VisibilityTimeout); + var cat = new QuestDbTagCatalog(c, b, wal); + return (new QuestDbTagProvisioner(c, b, cat, wal), c, cat); + } + + private static async Task<(byte Type, string? Eu, string? Desc)> TagAsync(QuestDbConnection c, string name) + { + await using var conn = await c.OpenAsync(default); + await using var cmd = c.CreateCommand(conn, $"SELECT data_type, eng_unit, description FROM {c.TagTable} WHERE name = $1"); + QuestDbConnection.BindText(cmd, name); + await using var r = await cmd.ExecuteReaderAsync(); + (await r.ReadAsync()).ShouldBeTrue(); + return (QuestDbConnection.ReadKind(r, 0), r.IsDBNull(1) ? null : r.GetString(1), r.IsDBNull(2) ? null : r.GetString(2)); + } + + private static async Task CountAsync(QuestDbConnection c) + { + await using var conn = await c.OpenAsync(default); + await using var cmd = c.CreateCommand(conn, $"SELECT count() FROM {c.TagTable}"); + return Convert.ToInt64(await cmd.ExecuteScalarAsync()); + } + + [Fact] + public async Task Upsert_creates_then_changes_DataType_on_rerun_and_every_type_is_accepted() + { + fx.SkipIfUnavailable(); + var (p, c, _) = Make(); + var r1 = await p.EnsureTagsAsync([new("a", DriverDataType.Boolean, "degC", "desc"), new("s", DriverDataType.String, null, null)], CancellationToken.None); + r1.ShouldBe(new HistorianProvisionResult(2, 2, 0, 0)); + (await TagAsync(c, "a")).ShouldBe(((byte)DriverDataType.Boolean, "degC", "desc")); + var r2 = await p.EnsureTagsAsync([new("a", DriverDataType.Float64, null, null)], CancellationToken.None); + r2.ShouldBe(new HistorianProvisionResult(1, 1, 0, 0)); + (await TagAsync(c, "a")).ShouldBe(((byte)DriverDataType.Float64, "degC", "desc")); // null keeps stored metadata + (await CountAsync(c)).ShouldBe(2); // upsert by name, not a second row + } + + [Fact] + public async Task A_request_with_values_replaces_only_the_fields_it_carries() + { + fx.SkipIfUnavailable(); + var (p, c, _) = Make(); + await p.EnsureTagsAsync([new("m", DriverDataType.Int32, "A", "first")], CancellationToken.None); + await p.EnsureTagsAsync([new("m", DriverDataType.Int32, "B", null)], CancellationToken.None); + (await TagAsync(c, "m")).ShouldBe(((byte)DriverDataType.Int32, "B", "first")); + } + + [Fact] + public async Task Duplicate_names_and_bad_names_are_tallied_not_thrown() + { + fx.SkipIfUnavailable(); + var (p, _, _) = Make(); + var r = await p.EnsureTagsAsync([ + new("d", DriverDataType.Int32, null, null), new("d", DriverDataType.Int64, null, null), + new("", DriverDataType.Int32, null, null), new(new string('x', 401), DriverDataType.Int32, null, null)], CancellationToken.None); + r.ShouldBe(new HistorianProvisionResult(4, 2, 0, 2)); + } + + [Fact] + public async Task More_tags_than_one_chunk_all_land() + { + fx.SkipIfUnavailable(); + var (p, c, _) = Make(); + var reqs = Enumerable.Range(0, 1000).Select(i => new HistorianTagProvisionRequest($"bulk{i}", DriverDataType.Float32, null, null)).ToList(); + (await p.EnsureTagsAsync(reqs, CancellationToken.None)).ShouldBe(new HistorianProvisionResult(1000, 1000, 0, 0)); + (await CountAsync(c)).ShouldBe(1000); + } + + [Fact] + public async Task A_later_catalog_Ensure_of_a_provisioned_tag_does_not_wipe_its_metadata() + { + fx.SkipIfUnavailable(); + var (p, c, cat) = Make(); + await p.EnsureTagsAsync([new("k", DriverDataType.Float64, "kg", "mass")], CancellationToken.None); + (await cat.ExistsAsync("k", CancellationToken.None)).ShouldBeTrue(); + await cat.EnsureAsync("k", 1, CancellationToken.None); + (await TagAsync(c, "k")).ShouldBe(((byte)DriverDataType.Float64, "kg", "mass")); + } + + [Fact] + public async Task A_connection_failure_is_a_Failed_tally_not_an_exception() + { + fx.SkipIfUnavailable(); + var (p, _, _) = Make(connectionString: "Host=localhost;Port=1;Username=admin;Password=quest;Database=qdb;ServerCompatibilityMode=NoTypeLoading;Timeout=2"); + (await p.EnsureTagsAsync([new("x", DriverDataType.Int32, null, null)], CancellationToken.None)).ShouldBe(new HistorianProvisionResult(1, 0, 0, 1)); + } +} diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/WriteTests.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/WriteTests.cs new file mode 100644 index 00000000..3f8a40f7 --- /dev/null +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/WriteTests.cs @@ -0,0 +1,221 @@ +using Shouldly; +using Xunit; +using ZB.MOM.WW.OtOpcUa.Core.Abstractions; +using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common; + +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests; + +[Collection(QuestDbCollection.Name)] +public sealed class WriteTests(QuestDbFixture fx) +{ + // µs-aligned (ticks % 10 == 0) so "same µs" offsets of a few ticks stay inside one 10-tick window. + private static readonly DateTime T0 = new(2026, 10, 9, 1, 2, 3, DateTimeKind.Utc); + private static readonly DateTime ServerNow = new(2026, 10, 9, 12, 0, 0, DateTimeKind.Utc); + + private (QuestDbHistorianValueWriter W, QuestDbConnection C) Make(string? prefix = null, string? connectionString = null) + { + var o = fx.Options(prefix ?? fx.UniquePrefix()); + if (connectionString is not null) o.ConnectionString = connectionString; + var c = new QuestDbConnection(o); + var b = new QuestDbSchemaBootstrapper(c, o); + var wal = new QuestDbWalVisibility(c, o.VisibilityTimeout); + return (new QuestDbHistorianValueWriter(c, new QuestDbTagCatalog(c, b, wal), wal, utcNow: () => ServerNow), c); + } + + private static async Task> RowsAsync( + QuestDbConnection c, string tag) + { + await using var conn = await c.OpenAsync(default); + await using var cmd = c.CreateCommand(conn, + $"SELECT ts, kind, num_value, int_value, str_value, status, deleted, server_ts FROM {c.SampleTable} WHERE tag = $1 ORDER BY ts"); + QuestDbConnection.BindText(cmd, tag); + var l = new List<(DateTime, byte, double?, long?, string?, long, bool, DateTime)>(); + await using var r = await cmd.ExecuteReaderAsync(); + while (await r.ReadAsync()) + l.Add((QuestDbConnection.ReadUtc(r, 0), QuestDbConnection.ReadKind(r, 1), r.IsDBNull(2) ? null : r.GetDouble(2), + r.IsDBNull(3) ? null : r.GetInt64(3), r.IsDBNull(4) ? null : r.GetString(4), r.GetInt64(5), r.GetBoolean(6), + QuestDbConnection.ReadUtc(r, 7))); + return l; + } + + private static async Task SeedTombstoneAsync(QuestDbConnection c, QuestDbWalVisibility wal, string tag, DateTime t) + { + await using (var conn = await c.OpenAsync(default)) + { + await using var cmd = c.CreateCommand(conn, + $"INSERT INTO {c.SampleTable} (ts, tag, kind, num_value, int_value, str_value, status, deleted, server_ts) VALUES ($1,$2,$3,NULL,NULL,NULL,$4,true,$5)"); + QuestDbConnection.BindTimestamp(cmd, t); + QuestDbConnection.BindText(cmd, tag); + QuestDbConnection.BindByte(cmd, 0); + QuestDbConnection.BindLong(cmd, 0); + QuestDbConnection.BindTimestamp(cmd, t); + await cmd.ExecuteNonQueryAsync(); + } + await wal.WaitAsync(c.SampleTable, default); + } + + [Fact] + public async Task All_six_kinds_round_trip_with_a_non_Good_status() + { + fx.SkipIfUnavailable(); + var (w, c) = Make(); + const uint uncertain = HistorianStatusCodes.UncertainDataSubNormal; + const uint bad = HistorianStatusCodes.BadOutOfRange; + var dt = new DateTime(2025, 5, 6, 7, 8, 9, DateTimeKind.Utc).AddTicks(1230); + HistorizationValue[] vals = + [ + new(T0, HistorianValue.FromDouble(1.5), uncertain), + new(T0.AddSeconds(1), HistorianValue.FromInt64(-77), uncertain), + new(T0.AddSeconds(2), HistorianValue.FromBoolean(true), uncertain), + new(T0.AddSeconds(3), HistorianValue.FromString("héllo"), uncertain), + new(T0.AddSeconds(4), HistorianValue.FromDateTime(dt), uncertain), + new(T0.AddSeconds(5), HistorianValue.Empty, bad), + ]; + (await w.WriteLiveValuesAsync("tag.kinds", vals, CancellationToken.None)).ShouldBeTrue(); + + var rows = await RowsAsync(c, "tag.kinds"); + rows.Count.ShouldBe(6); + for (var i = 0; i < 6; i++) + { + rows[i].T.ShouldBe(vals[i].TimestampUtc!.Value); + rows[i].Status.ShouldBe(SampleColumnMapper.ToLongStatus(vals[i].StatusCode)); + rows[i].Deleted.ShouldBeFalse(); + rows[i].ServerTs.ShouldBe(ServerNow); + SampleColumnMapper.FromColumns(rows[i].Kind, rows[i].N, rows[i].I, rows[i].S).ShouldBe(vals[i].Value); + } + } + + [Fact] + public async Task A_redelivered_batch_leaves_one_row_and_the_first_value() + { + fx.SkipIfUnavailable(); + var (w, c) = Make(); + (await w.WriteLiveValuesAsync("tag.redeliver", [new(T0, HistorianValue.FromDouble(1), 0)], CancellationToken.None)).ShouldBeTrue(); + (await w.WriteLiveValuesAsync("tag.redeliver", [new(T0, HistorianValue.FromDouble(2), 0)], CancellationToken.None)).ShouldBeTrue(); + var rows = await RowsAsync(c, "tag.redeliver"); + rows.Count.ShouldBe(1); + rows[0].N.ShouldBe(1d); + } + + [Fact] + public async Task Redelivery_straight_after_the_write_always_sees_it_because_the_writer_waits_for_WAL_visibility() + { + fx.SkipIfUnavailable(); + var (w, c) = Make(); + // Without the visibility wait ~35 % of immediate re-reads miss the row (T0 Q4) and the second value would replace the first. + for (var i = 0; i < 40; i++) + { + var t = T0.AddSeconds(i); + (await w.WriteLiveValuesAsync("tag.rw", [new(t, HistorianValue.FromDouble(1), 0)], CancellationToken.None)).ShouldBeTrue(); + (await w.WriteLiveValuesAsync("tag.rw", [new(t, HistorianValue.FromDouble(2), 0)], CancellationToken.None)).ShouldBeTrue(); + } + var rows = await RowsAsync(c, "tag.rw"); + rows.Count.ShouldBe(40); + rows.ShouldAllBe(r => r.N == 1d); + } + + [Fact] + public async Task A_tombstoned_key_is_not_resurrected_by_redelivery() + { + fx.SkipIfUnavailable(); + var (w, c) = Make(); + var wal = new QuestDbWalVisibility(c, TimeSpan.FromSeconds(5)); + (await w.WriteLiveValuesAsync("tag.tomb", [new(T0, HistorianValue.FromDouble(9), 0)], CancellationToken.None)).ShouldBeTrue(); + await SeedTombstoneAsync(c, wal, "tag.tomb", T0.AddSeconds(1)); + + // Redeliver the tombstoned key plus a fresh neighbour. + (await w.WriteLiveValuesAsync("tag.tomb", + [new(T0.AddSeconds(1), HistorianValue.FromDouble(5), 0), new(T0.AddSeconds(2), HistorianValue.FromDouble(6), 0)], + CancellationToken.None)).ShouldBeTrue(); + + var rows = await RowsAsync(c, "tag.tomb"); + rows.Count.ShouldBe(3); + rows.Single(r => r.T == T0.AddSeconds(1)).Deleted.ShouldBeTrue(); + rows.Single(r => r.T == T0.AddSeconds(2)).N.ShouldBe(6d); + } + + [Fact] + public async Task NaN_is_stored_as_Empty_with_BadOutOfRange() + { + fx.SkipIfUnavailable(); + var (w, c) = Make(); + (await w.WriteLiveValuesAsync("tag.nan", [new(T0, HistorianValue.FromDouble(double.NaN), 0)], CancellationToken.None)).ShouldBeTrue(); + var rows = await RowsAsync(c, "tag.nan"); + rows.Count.ShouldBe(1); + rows[0].Kind.ShouldBe((byte)HistorianValueKind.Empty); + rows[0].Status.ShouldBe((long)HistorianStatusCodes.BadOutOfRange); + rows[0].N.ShouldBeNull(); + } + + [Fact] + public async Task A_microsecond_collision_within_a_batch_keeps_the_first() + { + fx.SkipIfUnavailable(); + var (w, c) = Make(); + // +2 and +7 ticks stay inside one 10-tick (1 µs) window of the aligned T0. + (await w.WriteLiveValuesAsync("tag.us", + [new(T0.AddTicks(2), HistorianValue.FromDouble(1), 0), new(T0.AddTicks(7), HistorianValue.FromDouble(2), 0)], + CancellationToken.None)).ShouldBeTrue(); + var rows = await RowsAsync(c, "tag.us"); + rows.Count.ShouldBe(1); + rows[0].N.ShouldBe(1d); + rows[0].T.ShouldBe(T0); + } + + [Fact] + public async Task Samples_before_1970_are_dropped_and_the_rest_still_land() + { + fx.SkipIfUnavailable(); + var (w, c) = Make(); + var old = new DateTime(1960, 1, 1, 0, 0, 0, DateTimeKind.Utc); + (await w.WriteLiveValuesAsync("tag.old", + [new(old, HistorianValue.FromDouble(1), 0), new(T0, HistorianValue.FromDouble(2), 0)], CancellationToken.None)).ShouldBeTrue(); + w.DroppedPre1970.ShouldBe(1); + var rows = await RowsAsync(c, "tag.old"); + rows.Count.ShouldBe(1); + rows[0].T.ShouldBe(T0); + + // A batch of nothing but unstorable samples succeeds (nothing to retry) and stores nothing. + (await w.WriteLiveValuesAsync("tag.old2", [new(old, HistorianValue.FromDouble(1), 0)], CancellationToken.None)).ShouldBeTrue(); + w.DroppedPre1970.ShouldBe(2); + (await RowsAsync(c, "tag.old2")).ShouldBeEmpty(); + } + + [Fact] + public async Task A_batch_larger_than_one_insert_chunk_lands_completely() + { + fx.SkipIfUnavailable(); + var (w, c) = Make(); + var vals = Enumerable.Range(0, 2500).Select(i => new HistorizationValue(T0.AddMilliseconds(i), HistorianValue.FromInt64(i), 0)).ToArray(); + (await w.WriteLiveValuesAsync("tag.big", vals, CancellationToken.None)).ShouldBeTrue(); + var rows = await RowsAsync(c, "tag.big"); + rows.Count.ShouldBe(2500); + rows[1999].I.ShouldBe(1999); + } + + [Fact] + public async Task An_over_long_tag_name_returns_false() + { + fx.SkipIfUnavailable(); + var (w, _) = Make(); + (await w.WriteLiveValuesAsync(new string('x', 401), [new(T0, HistorianValue.FromDouble(1), 0)], CancellationToken.None)).ShouldBeFalse(); + } + + [Fact] + public async Task A_stopped_port_returns_false_without_throwing() + { + fx.SkipIfUnavailable(); + var (w, _) = Make(connectionString: "Host=localhost;Port=1;Username=admin;Password=quest;Database=qdb;ServerCompatibilityMode=NoTypeLoading;Timeout=2"); + (await w.WriteLiveValuesAsync("x", [new(T0, HistorianValue.FromDouble(1), 0)], CancellationToken.None)).ShouldBeFalse(); + } + + [Fact] + public async Task A_cancelled_write_returns_false_without_throwing() + { + fx.SkipIfUnavailable(); + var (w, _) = Make(); + using var cts = new CancellationTokenSource(); + cts.Cancel(); + (await w.WriteLiveValuesAsync("x", [new(T0, HistorianValue.FromDouble(1), 0)], cts.Token)).ShouldBeFalse(); + } +} From 5c0671e5b51a6dd8154f369a6142dd1f0719c14a Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Sun, 11 Oct 2026 02:12:09 -0400 Subject: [PATCH 13/23] fix(historian): per-tag update lock + clustered existence reads (T6 review) Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe --- .../QuestDbHistorianDataUpdater.cs | 78 ++++++++++++++++--- .../UpdateTests.cs | 43 ++++++++++ 2 files changed, 109 insertions(+), 12 deletions(-) diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianDataUpdater.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianDataUpdater.cs index c818e8b0..27e4f416 100644 --- a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianDataUpdater.cs +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianDataUpdater.cs @@ -15,6 +15,10 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb; /// "Exists" means present and not deleted, so Insert over a tombstone succeeds (GoodEntryInserted) while /// Replace and DeleteAtTime treat a tombstone as no entry. Existence is read with one range query and filtered /// client-side (never an IN list). +/// Concurrency: calls are serialized per tag inside this instance (64 striped async locks held from +/// the existence read through the visibility wait), so concurrent Insert/Replace/Delete on one tag cannot +/// both act on a stale read. This does NOT cover the live writer racing on the same key, nor a second node; +/// that residual is accepted. /// Pre-1970 (RC1): QuestDB cannot store such a designated timestamp. Insert/Replace/Update of such an /// item reports BadOutOfRange and sends nothing (a null/MinValue timestamp is still /// BadInvalidTimestamp); DeleteAtTime reports BadNoEntryExists; a DeleteRaw window may start before @@ -27,7 +31,13 @@ internal sealed class QuestDbHistorianDataUpdater : IHistorianDataUpdater /// Rows per INSERT and per DeleteRaw page (T0 Q2: 9,000 parameters, far under the 32,767 limit). internal const int BatchSize = 1000; + /// Request times further apart than this are read with separate range queries (I-2). + internal static readonly TimeSpan ExistenceClusterGap = TimeSpan.FromMinutes(1); + private const long ResolutionTicks = 10; + private const int LockStripes = 64; + private readonly SemaphoreSlim[] _locks = Enumerable.Range(0, LockStripes).Select(_ => new SemaphoreSlim(1, 1)).ToArray(); + private long _liveRowsRead; private readonly QuestDbConnection _conn; private readonly QuestDbTagCatalog _tags; private readonly QuestDbWalVisibility _wal; @@ -47,6 +57,41 @@ internal sealed class QuestDbHistorianDataUpdater : IHistorianDataUpdater _log = log ?? NullLogger.Instance; } + /// Rows returned by existence reads so far (test/diagnostic counter). + internal long LiveRowsRead => Interlocked.Read(ref _liveRowsRead); + + /// Splits times into clusters wherever consecutive sorted times are further apart than . + /// Request times (any order). + /// Largest gap kept inside one cluster. + /// Inclusive [Lo, Hi] ranges, ascending. + internal static List<(DateTime Lo, DateTime Hi)> Cluster(IReadOnlyList times, TimeSpan gap) + { + var result = new List<(DateTime, DateTime)>(); + if (times.Count == 0) return result; + var sorted = times.OrderBy(t => t).ToList(); + var lo = sorted[0]; + var prev = lo; + foreach (var t in sorted.Skip(1)) + { + if (t - prev > gap) { result.Add((lo, prev)); lo = t; } + prev = t; + } + result.Add((lo, prev)); + return result; + } + + private async Task LockAsync(string tag, CancellationToken ct) + { + var sem = _locks[(int)((uint)StringComparer.Ordinal.GetHashCode(tag) % LockStripes)]; + await sem.WaitAsync(ct).ConfigureAwait(false); + return new Releaser(sem); + } + + private sealed class Releaser(SemaphoreSlim sem) : IDisposable + { + public void Dispose() => sem.Release(); + } + /// public async Task> UpdateDataAsync(string tag, HistoryUpdateMode mode, IReadOnlyList values, CancellationToken ct) @@ -73,6 +118,7 @@ internal sealed class QuestDbHistorianDataUpdater : IHistorianDataUpdater } else { + using var _ = await LockAsync(tag, ct).ConfigureAwait(false); var known = mode == HistoryUpdateMode.Replace ? await _tags.ExistsAsync(tag, ct).ConfigureAwait(false) : await EnsureAsync(tag, plan.Winners[0].Row.Value.Kind, ct).ConfigureAwait(false); @@ -93,6 +139,7 @@ internal sealed class QuestDbHistorianDataUpdater : IHistorianDataUpdater if (!await _tags.ExistsAsync(tag, ct).ConfigureAwait(false)) return HistoryUpdatePlanner.DeleteRawResult(tagKnown: false, 0); + using var _ = await LockAsync(tag, ct).ConfigureAwait(false); var cursor = QuestDbConnection.Round(startUtc); var end = QuestDbConnection.Round(endUtc); long total = 0; @@ -142,7 +189,8 @@ internal sealed class QuestDbHistorianDataUpdater : IHistorianDataUpdater if (candidates.Count == 0 || !await _tags.ExistsAsync(tag, ct).ConfigureAwait(false)) return HistoryUpdatePlanner.DeleteAtTimeResults(rounded, deleted, tagKnown: false); - var live = await ReadLiveAsync(tag, candidates.Min(), candidates.Max(), ct).ConfigureAwait(false); + using var _ = await LockAsync(tag, ct).ConfigureAwait(false); + var live = await ReadLiveAsync(tag, candidates.Select(t => new DateTime(t, DateTimeKind.Utc)).ToList(), ct).ConfigureAwait(false); var toDelete = candidates.Where(live.Contains).OrderBy(x => x).ToList(); foreach (var chunk in toDelete.Chunk(BatchSize)) await WriteTombstonesAsync(tag, chunk.Select(t => new DateTime(t, DateTimeKind.Utc)).ToList(), ct).ConfigureAwait(false); @@ -161,8 +209,7 @@ internal sealed class QuestDbHistorianDataUpdater : IHistorianDataUpdater List rows, CancellationToken ct) { // Key = the planner's tick (what Complete looks up); the stored time is Round(of the same row). - var live = await ReadLiveAsync(tag, rows.Min(r => QuestDbConnection.Round(r.TimeUtc).Ticks), - rows.Max(r => QuestDbConnection.Round(r.TimeUtc).Ticks), ct).ConfigureAwait(false); + var live = await ReadLiveAsync(tag, rows.Select(r => QuestDbConnection.Round(r.TimeUtc)).ToList(), ct).ConfigureAwait(false); var outcome = new Dictionary(); var toWrite = new List(); foreach (var r in rows) @@ -179,18 +226,25 @@ internal sealed class QuestDbHistorianDataUpdater : IHistorianDataUpdater return outcome; } - /// Ticks of the live (present, not deleted) rows with ts in [minTicks, maxTicks]: one range query. - private async Task> ReadLiveAsync(string tag, long minTicks, long maxTicks, CancellationToken ct) + /// Ticks of the live (present, not deleted) rows near the given times: one range query per cluster (I-2). + private async Task> ReadLiveAsync(string tag, IReadOnlyList times, CancellationToken ct) { var live = new HashSet(); await using var conn = await _conn.OpenAsync(ct).ConfigureAwait(false); - await using var cmd = _conn.CreateCommand(conn, - $"SELECT ts FROM {_conn.SampleTable} WHERE tag = $1 AND ts >= $2 AND ts <= $3 AND NOT deleted"); - QuestDbConnection.BindText(cmd, tag); - QuestDbConnection.BindTimestamp(cmd, new DateTime(minTicks, DateTimeKind.Utc)); - QuestDbConnection.BindTimestamp(cmd, new DateTime(maxTicks, DateTimeKind.Utc)); - await using var rd = await cmd.ExecuteReaderAsync(ct).ConfigureAwait(false); - while (await rd.ReadAsync(ct).ConfigureAwait(false)) live.Add(QuestDbConnection.ReadUtc(rd, 0).Ticks); + foreach (var (lo, hi) in Cluster(times, ExistenceClusterGap)) + { + await using var cmd = _conn.CreateCommand(conn, + $"SELECT ts FROM {_conn.SampleTable} WHERE tag = $1 AND ts >= $2 AND ts <= $3 AND NOT deleted"); + QuestDbConnection.BindText(cmd, tag); + QuestDbConnection.BindTimestamp(cmd, lo); + QuestDbConnection.BindTimestamp(cmd, hi); + await using var rd = await cmd.ExecuteReaderAsync(ct).ConfigureAwait(false); + while (await rd.ReadAsync(ct).ConfigureAwait(false)) + { + live.Add(QuestDbConnection.ReadUtc(rd, 0).Ticks); + Interlocked.Increment(ref _liveRowsRead); + } + } return live; } diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/UpdateTests.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/UpdateTests.cs index c77f3d64..0f721065 100644 --- a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/UpdateTests.cs +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/UpdateTests.cs @@ -232,4 +232,47 @@ public sealed class UpdateTests(QuestDbFixture fx) .ShouldBe([HistorianStatusCodes.GoodEntryReplaced], $"iteration {i}: the insert was not visible"); } } + + [Fact] + public async Task Concurrent_inserts_of_the_same_new_key_yield_exactly_one_GoodEntryInserted() + { + fx.SkipIfUnavailable(); + var rig = MakeRig(fx); + for (var i = 0; i < 20; i++) // loop so the race shows reliably without the lock + { + var t = T0.AddSeconds(i); + var codes = await Task.WhenAll( + Task.Run(() => rig.Updater.UpdateDataAsync("race", HistoryUpdateMode.Insert, [V(t, 1)], default)), + Task.Run(() => rig.Updater.UpdateDataAsync("race", HistoryUpdateMode.Insert, [V(t, 2)], default))); + codes.Select(c => c[0]).OrderBy(c => c).ToArray().ShouldBe( + new[] { HistorianStatusCodes.GoodEntryInserted, HistorianStatusCodes.BadEntryExists }, $"iteration {i}"); + } + } + + [Fact] + public void Cluster_splits_where_the_gap_exceeds_the_limit() + { + var gap = QuestDbHistorianDataUpdater.ExistenceClusterGap; + var c = QuestDbHistorianDataUpdater.Cluster( + [T0.AddHours(2), T0, T0.AddSeconds(30), T0.AddHours(1)], gap); + c.ShouldBe([(T0, T0.AddSeconds(30)), (T0.AddHours(1), T0.AddHours(1)), (T0.AddHours(2), T0.AddHours(2))]); + QuestDbHistorianDataUpdater.Cluster([], gap).ShouldBeEmpty(); + } + + [Fact] + public async Task Existence_reads_are_bounded_when_request_times_are_far_apart() + { + fx.SkipIfUnavailable(); + var rig = MakeRig(fx); + // 1 Hz seed across 2 h = 7,200 live rows; the request touches 3 instants 1 h apart. + var seed = Enumerable.Range(0, 7200).Select(i => V(T0.AddSeconds(i), i)).ToList(); + await rig.Updater.UpdateDataAsync("sparse", HistoryUpdateMode.Insert, seed, default); + var before = rig.Updater.LiveRowsRead; + var codes = await rig.Updater.UpdateDataAsync("sparse", HistoryUpdateMode.Replace, + [V(T0, 1), V(T0.AddHours(1), 2), V(T0.AddHours(2).AddSeconds(-1), 3), V(T0.AddHours(1).AddMilliseconds(500), 4)], default); + codes.ShouldBe([HistorianStatusCodes.GoodEntryReplaced, HistorianStatusCodes.GoodEntryReplaced, + HistorianStatusCodes.GoodEntryReplaced, HistorianStatusCodes.BadNoEntryExists]); + // 3 clusters of a single instant each (T0+1h and +1h.5s are one cluster of 2 instants, 1 live row) + (rig.Updater.LiveRowsRead - before).ShouldBeLessThanOrEqualTo(3); + } } From c4b4b2e2bca5545db8cbafab3e48bb2acf60deaa Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Sun, 11 Oct 2026 02:13:00 -0400 Subject: [PATCH 14/23] feat(historian): QuestDB reads and aggregates (pushed down, shared evaluation) QuestDbHistorianDataSource serves Raw (LIMIT $n, no continuation, R8), AtTime (chunked LIMIT 1 neighbour seeks, <=50 times per statement), Events and Processed (one statement per node, bucket = (cast(ts AS LONG)*10 - start)/interval, shared BucketPlan/AggregateEvaluator/AtTimeEvaluator). Every sample read filters NOT deleted; every row goes through StoredSample.FromStored; RC2 long.MinValue read repair. QuestDbConformanceBackend (seed/read-back) for the conformance suite; writer/updater factories are T7 placeholders. Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe --- .../QuestDbAggregateQueries.cs | 122 +++++ .../QuestDbHistorianDataSource.cs | 468 ++++++++++++++++++ .../QuestDbConformanceBackend.cs | 203 ++++++++ .../ReadTests.cs | 266 ++++++++++ 4 files changed, 1059 insertions(+) create mode 100644 src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbAggregateQueries.cs create mode 100644 src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianDataSource.cs create mode 100644 tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/QuestDbConformanceBackend.cs create mode 100644 tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/ReadTests.cs diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbAggregateQueries.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbAggregateQueries.cs new file mode 100644 index 00000000..afe02efa --- /dev/null +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbAggregateQueries.cs @@ -0,0 +1,122 @@ +using System.Globalization; +using ZB.MOM.WW.OtOpcUa.Core.Abstractions; +using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common; + +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb; + +/// +/// Push-down QuestDB SQL for processed (aggregate) reads (spec §3 "Processed", T0 Q8-confirmed forms), one query +/// per , mirroring the SQL Server backend's SqlAggregateQueries family split and +/// column order. Every query takes five positional parameters: +/// +/// $1 — the (rounded) window start in 100 ns ticks since the Unix epoch +/// (, Bigint); +/// $2 — the bucket width in ticks (, Bigint); +/// $3 — the tag (Varchar); +/// $4/$5 — the forward window [start, end) (TimestampTz via +/// ). +/// +/// Each returns one row per non-empty bucket; the caller fills the empty ones. +/// +/// +/// +/// Bucketing. b = (cast(ts AS LONG) * 10 − $1) / $2. cast(ts AS LONG) is µs since 1970, so +/// × 10 is ticks since 1970; both it and $1 are µs-aligned (R6), so the integer division is +/// start-inclusive / end-exclusive to the tick, exactly like SQL Server's. Every row is at or after the start, +/// so the dividend is never negative (no truncation-toward-zero surprise), and nothing overflows before 9999. +/// SAMPLE BY is deliberately not used: its alignment rules differ from ours. +/// +/// +/// Sample classes (S6). The status is stored as the unsigned 32-bit code in a LONG, so +/// "not Bad" is status < 0x80000000 (not T-SQL's sign test). usable = not Bad AND +/// kind <> 0 (an Empty sample is an outage marker and counts as Bad whatever its code); +/// good = usable AND status < 0x40000000; bad = not usable; non-numeric = usable +/// String/DateTime. The literals are generated from and . +/// +/// Common result columns (ordinals 0–5): b, total_count, usable_count, good_count, bad_count, +/// non_numeric_count. Family columns follow from . +/// +internal static class QuestDbAggregateQueries +{ + /// Ordinal of the first family-specific column. + public const int FirstFamilyColumn = 6; + + /// status < SeverityBadBit, as a decimal LONG literal. + internal static readonly string NotBadLimit = StatusBits.SeverityBadBit.ToString(CultureInfo.InvariantCulture); + + /// status < SeverityUncertainBit, as a decimal LONG literal. + internal static readonly string GoodLimit = StatusBits.SeverityUncertainBit.ToString(CultureInfo.InvariantCulture); + + /// The kind values of String and DateTime: usable but not numeric. + internal static readonly string NonNumericKinds = + ((byte)HistorianValueKind.String).ToString(CultureInfo.InvariantCulture) + ", " + + ((byte)HistorianValueKind.DateTime).ToString(CultureInfo.InvariantCulture); + + private static readonly string EmptyKind = ((byte)HistorianValueKind.Empty).ToString(CultureInfo.InvariantCulture); + + /// + /// Average / Minimum / Maximum / Total / Count / Range / PercentGood / PercentBad. + /// Family columns: 6 avg_value, 7 min_value, 8 max_value, 9 sum_value (DOUBLE, NULL when no usable sample). + /// + /// Prefixed sample table name. + /// The SQL text. + public static string Simple(string sample) => + "SELECT b, " + CommonCounts() + ", " + + "avg(CASE WHEN u = 1 THEN num_value END) AS avg_value, " + + "min(CASE WHEN u = 1 THEN num_value END) AS min_value, " + + "max(CASE WHEN u = 1 THEN num_value END) AS max_value, " + + "sum(CASE WHEN u = 1 THEN num_value END) AS sum_value " + + "FROM (" + Classified(sample) + ") GROUP BY b ORDER BY b"; + + /// + /// Start / End / Delta: the first and last usable sample of each bucket by ts. + /// Family columns: 6..10 first sample (kind, num_value, int_value, str_value, ts), 11..15 last + /// sample (same order). A bucket without a usable sample has NULL ts but kind = 0 (BYTE has no NULL), + /// so the reader keys absence on the NULL time. + /// + /// Prefixed sample table name. + /// The SQL text. + public static string StartEnd(string sample) => + "WITH s AS (" + Classified(sample) + "), " + + "st AS (SELECT b, " + CommonCounts() + " FROM s GROUP BY b), " + + "f AS (SELECT b, kind, num_value, int_value, str_value, ts, " + + "row_number() OVER (PARTITION BY b ORDER BY ts ASC) AS rf, " + + "row_number() OVER (PARTITION BY b ORDER BY ts DESC) AS rl FROM s WHERE u = 1) " + + "SELECT st.*, " + + "x.kind, x.num_value, x.int_value, x.str_value, x.ts, " + + "y.kind, y.num_value, y.int_value, y.str_value, y.ts " + + "FROM st LEFT JOIN (SELECT * FROM f WHERE rf = 1) x ON x.b = st.b " + + "LEFT JOIN (SELECT * FROM f WHERE rl = 1) y ON y.b = st.b ORDER BY st.b"; + + /// + /// MinimumActualTime / MaximumActualTime: the usable sample with the smallest (largest) num_value, ties + /// broken by the earliest ts. Family columns: 6 num_value, 7 ts (NULL when the bucket has no usable + /// numeric sample). + /// + /// Prefixed sample table name. + /// True for MaximumActualTime. + /// The SQL text. + public static string ActualTime(string sample, bool maximum) => + "WITH s AS (" + Classified(sample) + "), " + + "st AS (SELECT b, " + CommonCounts() + " FROM s GROUP BY b), " + + "r AS (SELECT b, num_value, ts, " + + "row_number() OVER (PARTITION BY b ORDER BY num_value " + (maximum ? "DESC" : "ASC") + ", ts ASC) AS rn " + + "FROM s WHERE u = 1 AND num_value IS NOT NULL) " + + "SELECT st.*, r.num_value, r.ts " + + "FROM st LEFT JOIN (SELECT * FROM r WHERE rn = 1) r ON r.b = st.b ORDER BY st.b"; + + /// The bucket-index expression (exposed for tests). + internal const string BucketExpression = "(cast(ts AS LONG) * 10 - $1) / $2"; + + /// The classified subquery s: the window's live samples with bucket index and the usable/good flags. + private static string Classified(string sample) => + "SELECT " + BucketExpression + " AS b, ts, kind, num_value, int_value, str_value, " + + $"CASE WHEN status < {NotBadLimit} AND kind <> {EmptyKind} THEN 1 ELSE 0 END AS u, " + + $"CASE WHEN status < {GoodLimit} AND kind <> {EmptyKind} THEN 1 ELSE 0 END AS g " + + $"FROM {sample} WHERE tag = $3 AND ts >= $4 AND ts < $5 AND NOT deleted"; + + /// The five common count columns (after b). + private static string CommonCounts() => + "count() AS total_count, sum(u) AS usable_count, sum(g) AS good_count, count() - sum(u) AS bad_count, " + + $"sum(CASE WHEN u = 1 AND kind IN ({NonNumericKinds}) THEN 1 ELSE 0 END) AS non_numeric_count"; +} diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianDataSource.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianDataSource.cs new file mode 100644 index 00000000..f4b97539 --- /dev/null +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianDataSource.cs @@ -0,0 +1,468 @@ +using System.Globalization; +using System.Text; +using Microsoft.Extensions.Logging; +using Microsoft.Extensions.Logging.Abstractions; +using Npgsql; +using NpgsqlTypes; +using ZB.MOM.WW.OtOpcUa.Core.Abstractions; +using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common; + +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb; + +/// +/// QuestDB read side of the historian (spec §3 "Reads"): Raw (forward / reverse / point), AtTime (chunked +/// LIMIT 1 neighbour seeks, T0 Q9), Events, and Processed aggregates pushed down to QuestDB SQL +/// (, T0 Q8). Results match the SQL Server data source except that every +/// timestamp is at microsecond resolution. +/// +/// +/// +/// Timestamps (R6). Every inbound time is normalised with and every +/// is bound only through . Read bounds are not +/// clamped to the epoch (a pre-1970 bound is valid); only MinValue/MaxValue are nudged. Results +/// carry the rounded times, so an AtTime result is stamped at the rounded request time. +/// +/// +/// Reads never create tags (R7). A tag resolves only through ; +/// an unknown tag is a successful empty read (Raw), an all-BadNoData read (AtTime / Processed), never an +/// error. Every sample query filters NOT deleted, so a DeleteRaw/DeleteAtTime tombstone is invisible. +/// +/// +/// Values. Every row goes through (an Empty row with a non-Bad status +/// is served BadNoData). The value kind, not which column is populated, decides the value. QuestDB stores +/// in a LONG as NULL (its NULL sentinel), so an Int64 row whose +/// int_value is NULL and whose num_value is exactly −2^63 is served as +/// (ruling RC2). +/// +/// +/// Faults. As in the SQL Server data source: a database / bootstrap exception is recorded in +/// and rethrown, failing only that call. Client-request errors — an unsupported +/// aggregate () and a processed window over +/// intervals () — are thrown before +/// any I/O and are not counted as backend failures. +/// +/// +public sealed class QuestDbHistorianDataSource : IHistorianDataSource +{ + /// Most AtTime request times per statement (two UNION ALL branches each; QuestDB 10.0.1 refuses more than 126 branches). + public const int AtTimeChunkSize = 50; + + /// as a double (−2^63, exact): what an Int64 sample of that value leaves in num_value. + internal const double LongNullSentinelAsDouble = -9223372036854775808d; + + private const string SampleColumns = "ts, kind, num_value, int_value, str_value, status, server_ts"; + + private readonly QuestDbConnection _conn; + private readonly QuestDbSchemaBootstrapper _bootstrap; + private readonly QuestDbTagCatalog _catalog; + private readonly ILogger _log; + private readonly TimeProvider _time; + private readonly HistorianHealthCounters _health = new(); + + /// Creates the data source over the provider's shared connection, bootstrapper and tag catalog. + /// Connection helper. + /// Schema bootstrapper (shared with the other QuestDB seams). + /// Tag catalog (reads use only its read-only ). + /// Optional logger. + /// Clock for "now" (AtTime beyond the last sample, server timestamps); system by default. + internal QuestDbHistorianDataSource( + QuestDbConnection conn, QuestDbSchemaBootstrapper bootstrap, QuestDbTagCatalog catalog, ILogger? log = null, + TimeProvider? timeProvider = null) + { + _conn = conn ?? throw new ArgumentNullException(nameof(conn)); + _bootstrap = bootstrap ?? throw new ArgumentNullException(nameof(bootstrap)); + _catalog = catalog ?? throw new ArgumentNullException(nameof(catalog)); + _log = log ?? NullLogger.Instance; + _time = timeProvider ?? TimeProvider.System; + } + + // ------------------------------------------------------------------------------------------ Raw + + /// + /// + /// Forward (start < end): [start, end) ascending. Reverse (end < start): + /// (end, start] descending. start == end: the sample exactly at that (rounded) time. + /// 0 is "unbounded" (); larger values clamp. + /// At most top samples, ContinuationPoint always null (R8): the node manager pages by re-reading + /// from the last returned timestamp inclusive, which the (ts, tag) dedup key makes exact. + /// + public async Task ReadRawAsync( + string fullReference, DateTime startUtc, DateTime endUtc, uint maxValuesPerNode, CancellationToken cancellationToken) + { + ArgumentNullException.ThrowIfNull(fullReference); + var top = maxValuesPerNode == 0 ? int.MaxValue : (int)Math.Min(maxValuesPerNode, int.MaxValue); + try + { + var samples = new List(); + if (await ResolveAsync(fullReference, cancellationToken).ConfigureAwait(false)) + { + var start = QuestDbConnection.Round(startUtc); + var end = QuestDbConnection.Round(endUtc); + string where, order; + if (start == end) { where = "ts = $2 AND ts = $3"; order = "ASC"; } + else if (start < end) { where = "ts >= $2 AND ts < $3"; order = "ASC"; } + else { where = "ts > $2 AND ts <= $3"; order = "DESC"; } + var (lo, hi) = start <= end ? (start, end) : (end, start); + + await using var conn = await _conn.OpenAsync(cancellationToken).ConfigureAwait(false); + await using var cmd = _conn.CreateCommand(conn, + $"SELECT {SampleColumns} FROM {_conn.SampleTable} " + + $"WHERE tag = $1 AND {where} AND NOT deleted ORDER BY ts {order} LIMIT $4"); + QuestDbConnection.BindText(cmd, fullReference); + QuestDbConnection.BindTimestamp(cmd, lo); + QuestDbConnection.BindTimestamp(cmd, hi); + cmd.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Integer, Value = top }); + await using var reader = await cmd.ExecuteReaderAsync(cancellationToken).ConfigureAwait(false); + var now = _time.GetUtcNow().UtcDateTime; + while (await reader.ReadAsync(cancellationToken).ConfigureAwait(false)) + { + var s = ReadStored(reader, 0); + samples.Add(new DataValueSnapshot(s.Value.ToObject(), s.Status, s.TimeUtc, ReadServerTime(reader, 6, now))); + } + } + + RecordSuccess(); + return new HistoryReadResult(samples, ContinuationPoint: null); + } + catch (Exception ex) + { + RecordFailure(ex); + throw; + } + } + + // ------------------------------------------------------------------------------------------ AtTime + + /// + /// + /// Each distinct (rounded) requested time picks up its previous-or-equal and next live sample through two + /// LIMIT 1 seeks, UNION ALL-ed, at most times per statement (spec §3, + /// T0 Q9). then decides each value exactly as for SQL Server (exact hit, + /// no previous ⇒ BadNoData, inside an outage, held before an outage, linear, stepped, after now). The + /// result matches the request in length and order (duplicates included), each entry stamped at its rounded + /// time. Unknown tag ⇒ every entry BadNoData. + /// + public async Task ReadAtTimeAsync( + string fullReference, IReadOnlyList timestampsUtc, CancellationToken cancellationToken) + { + ArgumentNullException.ThrowIfNull(fullReference); + ArgumentNullException.ThrowIfNull(timestampsUtc); + try + { + var now = _time.GetUtcNow().UtcDateTime; + var rounded = timestampsUtc.Select(QuestDbConnection.Round).ToArray(); + var byTicks = new Dictionary(rounded.Length); + if (rounded.Length > 0 && await ResolveAsync(fullReference, cancellationToken).ConfigureAwait(false)) + { + var distinct = rounded.Distinct().ToList(); + await using var conn = await _conn.OpenAsync(cancellationToken).ConfigureAwait(false); + for (var offset = 0; offset < distinct.Count; offset += AtTimeChunkSize) + { + var chunk = distinct.GetRange(offset, Math.Min(AtTimeChunkSize, distinct.Count - offset)); + var prev = new StoredSample?[chunk.Count]; + var next = new StoredSample?[chunk.Count]; + var prevServer = new DateTime?[chunk.Count]; + + await using (var cmd = _conn.CreateCommand(conn, AtTimeSql(_conn.SampleTable, chunk.Count))) + { + QuestDbConnection.BindText(cmd, fullReference); + foreach (var t in chunk) QuestDbConnection.BindTimestamp(cmd, t); + await using var reader = await cmd.ExecuteReaderAsync(cancellationToken).ConfigureAwait(false); + while (await reader.ReadAsync(cancellationToken).ConfigureAwait(false)) + { + var k = Convert.ToInt32(reader.GetValue(0), CultureInfo.InvariantCulture); + var dir = Convert.ToInt32(reader.GetValue(1), CultureInfo.InvariantCulture); + var s = ReadStored(reader, 2); + if (dir == 0) + { + prev[k] = s; + prevServer[k] = reader.IsDBNull(8) ? null : QuestDbConnection.ReadUtc(reader, 8); + } + else + { + next[k] = s; + } + } + } + + for (var k = 0; k < chunk.Count; k++) + { + var t = chunk[k]; + var serverTime = prev[k] is { } ps && ps.TimeUtc == t ? prevServer[k] ?? now : now; + byTicks[t.Ticks] = AtTimeEvaluator.Evaluate(t, prev[k], next[k], serverTime, now); + } + } + } + + var result = new DataValueSnapshot[rounded.Length]; + for (var i = 0; i < result.Length; i++) + { + var t = rounded[i]; + result[i] = byTicks.TryGetValue(t.Ticks, out var v) + ? v + : new DataValueSnapshot(null, HistorianStatusCodes.BadNoData, t, now); + } + + RecordSuccess(); + return new HistoryReadResult(result, ContinuationPoint: null); + } + catch (Exception ex) + { + RecordFailure(ex); + throw; + } + } + + /// + /// The T0 Q9 statement for request times: $1 = tag, $(k+2) = time k. + /// Columns: req, dir (0 = previous-or-equal, 1 = next), then ts, kind, num_value, int_value, str_value, + /// status, server_ts. req/dir are generated integers, never user input. + /// + internal static string AtTimeSql(string sample, int count) + { + if (count is < 1 or > AtTimeChunkSize) throw new ArgumentOutOfRangeException(nameof(count)); + var sb = new StringBuilder(); + for (var k = 0; k < count; k++) + { + var p = "$" + (k + 2).ToString(CultureInfo.InvariantCulture); + var req = k.ToString(CultureInfo.InvariantCulture); + if (k > 0) sb.Append(" UNION ALL "); + sb.Append("SELECT * FROM (SELECT ").Append(req).Append(" AS req, 0 AS dir, ").Append(SampleColumns) + .Append(" FROM ").Append(sample).Append(" WHERE tag = $1 AND ts <= ").Append(p) + .Append(" AND NOT deleted ORDER BY ts DESC LIMIT 1)"); + sb.Append(" UNION ALL "); + sb.Append("SELECT * FROM (SELECT ").Append(req).Append(" AS req, 1 AS dir, ").Append(SampleColumns) + .Append(" FROM ").Append(sample).Append(" WHERE tag = $1 AND ts > ").Append(p) + .Append(" AND NOT deleted ORDER BY ts ASC LIMIT 1)"); + } + return sb.ToString(); + } + + // ------------------------------------------------------------------------------------------ Events + + /// + /// + /// [start, end) by event time (a reversed window is swapped), ordered by (ts, event_id) ascending. + /// <= 0 or ⇒ unbounded. Otherwise + /// maxEvents + 1 rows are fetched; when the extra row arrives it is dropped and the continuation point is + /// the UTF-8 of for that first not-returned event. + /// + public async Task ReadEventsAsync( + string? sourceName, DateTime startUtc, DateTime endUtc, int maxEvents, CancellationToken cancellationToken) + { + // No stored event can carry a source longer than the limit, so an over-long filter matches nothing. + if (sourceName is not null && sourceName.Length > HistorianTagNames.MaxLength) + return new HistoricalEventsResult([], null); + try + { + await _bootstrap.EnsureAsync(cancellationToken).ConfigureAwait(false); + var start = QuestDbConnection.Round(startUtc); + var end = QuestDbConnection.Round(endUtc); + var (lo, hi) = start <= end ? (start, end) : (end, start); + var bounded = maxEvents > 0 && maxEvents < int.MaxValue; + var sql = + $"SELECT event_id, source, ts, received_utc, message, severity FROM {_conn.EventTable} " + + "WHERE ts >= $1 AND ts < $2" + + (sourceName is null ? "" : " AND source = $3") + + " ORDER BY ts ASC, event_id ASC" + + (bounded ? (sourceName is null ? " LIMIT $3" : " LIMIT $4") : ""); + + await using var conn = await _conn.OpenAsync(cancellationToken).ConfigureAwait(false); + await using var cmd = _conn.CreateCommand(conn, sql); + QuestDbConnection.BindTimestamp(cmd, lo); + QuestDbConnection.BindTimestamp(cmd, hi); + if (sourceName is not null) QuestDbConnection.BindText(cmd, sourceName); + if (bounded) QuestDbConnection.BindLong(cmd, (long)maxEvents + 1); + + var events = new List(); + byte[]? continuation = null; + await using var reader = await cmd.ExecuteReaderAsync(cancellationToken).ConfigureAwait(false); + while (await reader.ReadAsync(cancellationToken).ConfigureAwait(false)) + { + var eventTime = QuestDbConnection.ReadUtc(reader, 2); + var e = new HistoricalEvent( + reader.IsDBNull(0) ? "" : reader.GetString(0), + reader.IsDBNull(1) ? null : reader.GetString(1), + eventTime, + reader.IsDBNull(3) ? eventTime : QuestDbConnection.ReadUtc(reader, 3), + reader.IsDBNull(4) ? null : reader.GetString(4), + HistoricalEventRows.ClampSeverity(reader.IsDBNull(5) ? 0 : reader.GetInt32(5))); + if (bounded && events.Count == maxEvents) + { + continuation = Encoding.UTF8.GetBytes(EventContinuation.Encode(e.EventTimeUtc, e.EventId)); + break; + } + events.Add(e); + } + + RecordSuccess(); + return new HistoricalEventsResult(events, continuation); + } + catch (Exception ex) + { + RecordFailure(ex); + throw; + } + } + + // ------------------------------------------------------------------------------------------ Processed + + /// + /// + /// The (rounded) window is planned by : bucket i = + /// [start + i·interval, start + (i+1)·interval), start-inclusive / end-exclusive, the last bucket truncated + /// at end, interval <= 0 ⇒ one bucket, start == end ⇒ none, a reversed window computed forward + /// and returned descending, at most buckets. One statement per node returns + /// the statistics of each non-empty bucket (bucket index + /// (cast(ts AS LONG)*10 − StartUnixTicks) / IntervalTicks, never SAMPLE BY); + /// produces each value and fills the + /// empty buckets with BadNoData. + /// + /// The aggregate is not one of the thirteen served. + /// The window holds more than intervals. + public async Task ReadProcessedAsync( + string fullReference, DateTime startUtc, DateTime endUtc, TimeSpan interval, + HistoryAggregateType aggregate, CancellationToken cancellationToken) + { + ArgumentNullException.ThrowIfNull(fullReference); + var family = AggregateFamilies.FamilyOf(aggregate) ?? throw new HistoryAggregateNotSupportedException(aggregate); + var plan = BucketPlan.Create(QuestDbConnection.Round(startUtc), QuestDbConnection.Round(endUtc), interval); + + try + { + var now = _time.GetUtcNow().UtcDateTime; + var evaluated = new Dictionary(); + if (plan.BucketCount > 0 && await ResolveAsync(fullReference, cancellationToken).ConfigureAwait(false)) + { + var sample = _conn.SampleTable; + var sql = family switch + { + AggregateFamily.Simple => QuestDbAggregateQueries.Simple(sample), + AggregateFamily.StartEnd => QuestDbAggregateQueries.StartEnd(sample), + _ => QuestDbAggregateQueries.ActualTime(sample, aggregate == HistoryAggregateType.MaximumActualTime), + }; + await using var conn = await _conn.OpenAsync(cancellationToken).ConfigureAwait(false); + await using var cmd = _conn.CreateCommand(conn, sql); + QuestDbConnection.BindLong(cmd, plan.StartUnixTicks); + QuestDbConnection.BindLong(cmd, plan.IntervalTicks); + QuestDbConnection.BindText(cmd, fullReference); + QuestDbConnection.BindTimestamp(cmd, plan.StartUtc); + QuestDbConnection.BindTimestamp(cmd, plan.EndUtc); + await using var reader = await cmd.ExecuteReaderAsync(cancellationToken).ConfigureAwait(false); + while (await reader.ReadAsync(cancellationToken).ConfigureAwait(false)) + { + var row = ReadBucket(reader, family); + evaluated[row.Bucket] = AggregateEvaluator.Evaluate(aggregate, row, plan.BucketStart(row.Bucket), now); + } + } + + var result = plan.Assemble(evaluated, bucketStart => AggregateEvaluator.NoData(bucketStart, now)); + + RecordSuccess(); + return new HistoryReadResult(result, ContinuationPoint: null); + } + catch (Exception ex) + { + RecordFailure(ex); + throw; + } + } + + private static BucketRow ReadBucket(NpgsqlDataReader reader, AggregateFamily family) + { + static double? D(NpgsqlDataReader r, int i) => r.IsDBNull(i) ? null : r.GetDouble(i); + static int N(NpgsqlDataReader r, int i) => r.IsDBNull(i) ? 0 : Convert.ToInt32(r.GetValue(i), CultureInfo.InvariantCulture); + // A bucket without a usable sample has NULL times but kind 0 (BYTE has no NULL): absence is keyed on the time. + static HistorianValue V(NpgsqlDataReader r, int i) => + r.IsDBNull(i + 4) + ? HistorianValue.Empty + : ToValue(QuestDbConnection.ReadKind(r, i), D(r, i + 1), r.IsDBNull(i + 2) ? null : r.GetInt64(i + 2), + r.IsDBNull(i + 3) ? null : r.GetString(i + 3)); + + var row = new BucketRow + { + Bucket = Convert.ToInt64(reader.GetValue(0), CultureInfo.InvariantCulture), + Total = N(reader, 1), + Usable = N(reader, 2), + Good = N(reader, 3), + Bad = N(reader, 4), + NonNumeric = N(reader, 5), + }; + var f = QuestDbAggregateQueries.FirstFamilyColumn; + switch (family) + { + case AggregateFamily.Simple: + row.Avg = D(reader, f); + row.Min = D(reader, f + 1); + row.Max = D(reader, f + 2); + row.Sum = D(reader, f + 3); + break; + case AggregateFamily.StartEnd: + row.First = V(reader, f); + row.Last = V(reader, f + 5); + break; + default: + row.Extreme = D(reader, f); + row.ExtremeTime = reader.IsDBNull(f + 1) ? null : QuestDbConnection.ReadUtc(reader, f + 1); + break; + } + return row; + } + + // ------------------------------------------------------------------------------------------ Shared + + /// + /// Reads ts, kind, num_value, int_value, str_value, status starting at and applies the + /// shared quality clamp (). + /// + private static StoredSample ReadStored(NpgsqlDataReader r, int o) + { + var value = ToValue( + QuestDbConnection.ReadKind(r, o + 1), + r.IsDBNull(o + 2) ? null : r.GetDouble(o + 2), + r.IsDBNull(o + 3) ? null : r.GetInt64(o + 3), + r.IsDBNull(o + 4) ? null : r.GetString(o + 4)); + // status is always written; a NULL (only possible for a hand-made row) is served as "no data", never Good. + var status = r.IsDBNull(o + 5) ? HistorianStatusCodes.BadNoData : SampleColumnMapper.FromLongStatus(r.GetInt64(o + 5)); + return StoredSample.FromStored(QuestDbConnection.ReadUtc(r, o), value, status); + } + + /// + /// Rebuilds a value from its stored columns via the shared , after the + /// RC2 repair: QuestDB reads a LONG holding back as NULL, so an Int64 row with + /// a NULL int_value and a num_value of exactly −2^63 is that value. + /// + internal static HistorianValue ToValue(byte kind, double? num, long? integer, string? str) + { + if (kind == (byte)HistorianValueKind.Int64 && integer is null && num == LongNullSentinelAsDouble) + integer = long.MinValue; + return SampleColumnMapper.FromColumns(kind, num, integer, str); + } + + private static DateTime ReadServerTime(NpgsqlDataReader r, int i, DateTime fallback) => + r.IsDBNull(i) ? fallback : QuestDbConnection.ReadUtc(r, i); + + /// Read-only tag lookup (R7; never creates a row); runs the schema bootstrap on a cache miss. + private Task ResolveAsync(string name, CancellationToken ct) => _catalog.ExistsAsync(name, ct); + + // ------------------------------------------------------------------------------------------ Health + + /// + public HistorianHealthSnapshot GetHealthSnapshot() => _health.Snapshot(); + + private void RecordSuccess() => _health.RecordSuccess(); + + private void RecordFailure(Exception ex) + { + // A cancelled read (deadline / client abort) is not a backend fault (the counters ignore it too). + if (ex is OperationCanceledException) return; + _health.RecordFailure(ex); + _log.LogDebug("QuestDB historian read failed ({ExceptionType})", ex.GetType().Name); + } + + /// + public void Dispose() + { + // Connections are per call (pooled by the shared NpgsqlDataSource, owned by the provider); nothing to release. + } +} diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/QuestDbConformanceBackend.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/QuestDbConformanceBackend.cs new file mode 100644 index 00000000..5dfc6f4c --- /dev/null +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/QuestDbConformanceBackend.cs @@ -0,0 +1,203 @@ +using System.Text; +using Npgsql; +using NpgsqlTypes; +using ZB.MOM.WW.OtOpcUa.Core.Abstractions; +using ZB.MOM.WW.OtOpcUa.Driver.Historian.Conformance; +using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common; + +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests; + +/// +/// The QuestDB historian as a conformance backend: a fresh unique table prefix per instance (dropped on dispose), +/// seeded by plain multi-row INSERTs (never through the writer or updater) followed by the WAL visibility wait. +/// Storage resolution is 1 µs (10 ticks); is , and +/// rounds the times it is given the same way before storing them. +/// +internal sealed class QuestDbConformanceBackend : IConformanceBackend +{ + private const int SeedBatchRows = 1000; + + private readonly QuestDbFixture _fx; + private readonly string _prefix; + + private QuestDbConformanceBackend(QuestDbFixture fx, string prefix, QuestDbHistorianOptions options, + QuestDbConnection conn, QuestDbSchemaBootstrapper boot) + { + _fx = fx; + _prefix = prefix; + Options = options; + Conn = conn; + Boot = boot; + Wal = new QuestDbWalVisibility(conn, options.VisibilityTimeout); + } + + /// The options of this instance's table set. + public QuestDbHistorianOptions Options { get; } + + /// The connection helper. + public QuestDbConnection Conn { get; } + + /// The schema bootstrapper (already run). + public QuestDbSchemaBootstrapper Boot { get; } + + /// The WAL visibility wait. + public QuestDbWalVisibility Wal { get; } + + /// Creates a backend over a fresh table prefix; skips the calling test when the gate is closed. + /// The shared fixture. + /// The backend. + public static async Task CreateAsync(QuestDbFixture fx) => await CreateConcreteAsync(fx); + + /// As , typed (for QuestDB-specific tests that need the connection). + /// The shared fixture. + /// The backend. + public static async Task CreateConcreteAsync(QuestDbFixture fx) + { + fx.SkipIfUnavailable(); + var prefix = fx.UniquePrefix(); + var o = fx.Options(prefix); + var conn = new QuestDbConnection(o); + var boot = new QuestDbSchemaBootstrapper(conn, o); + await boot.EnsureAsync(CancellationToken.None); + return new QuestDbConformanceBackend(fx, prefix, o, conn, boot); + } + + /// + public long ResolutionTicks => 10; + + /// + public DateTime Round(DateTime t) => QuestDbConnection.Round(t); + + /// A fresh tag catalog (fresh positive cache) over this table set. + public QuestDbTagCatalog NewCatalog() => new(Conn, Boot, Wal); + + /// + public IHistorianDataSource CreateDataSource(TimeProvider? clock = null) => + new QuestDbHistorianDataSource(Conn, Boot, NewCatalog(), timeProvider: clock); + + /// + /// TODO(T7): build the T4 QuestDbHistorianValueWriter here once T4 is merged. + public IHistorianValueWriter CreateValueWriter(Func? utcNow = null) => + throw new NotSupportedException("TODO(T7): the QuestDB value writer (T4) is not merged into this branch yet."); + + /// + /// TODO(T7): build the T6 QuestDbHistorianDataUpdater here once T6 is merged. + public IHistorianDataUpdater CreateUpdater() => + throw new NotSupportedException("TODO(T7): the QuestDB history updater (T6) is not merged into this branch yet."); + + /// + /// + /// Times are rounded to the microsecond () before they are stored; the tag row is registered + /// through the catalog (data type 0, as the SQL Server seed does); every row is live (deleted = false); + /// server_ts = rounded time + 1 s. Waits for WAL visibility before returning. + /// + public async Task SeedAsync(string tag, params (DateTime T, HistorianValue V, uint Status)[] samples) + { + await NewCatalog().EnsureAsync(tag, 0, CancellationToken.None); + await InsertRowsAsync(tag, samples.Select(s => (Round(s.T), s.V, s.Status, Deleted: false)).ToArray()); + } + + /// + /// Inserts raw rows (the tag must already be registered when reads should find it), including tombstones, then + /// waits for WAL visibility. A tombstone re-inserted at an existing (ts, tag) replaces that row (DEDUP). + /// + /// The tag. + /// Rows: time (already rounded by the caller or not — it is bound through BindTimestamp), value, status, deleted. + public async Task InsertRowsAsync(string tag, (DateTime T, HistorianValue V, uint Status, bool Deleted)[] rows) + { + if (rows.Length == 0) return; + await using (var c = await Conn.OpenAsync(CancellationToken.None)) + { + for (var offset = 0; offset < rows.Length; offset += SeedBatchRows) + { + var batch = rows.Skip(offset).Take(SeedBatchRows).ToArray(); + var sql = new StringBuilder( + $"INSERT INTO {Conn.SampleTable} (ts, tag, kind, num_value, int_value, str_value, status, deleted, server_ts) VALUES "); + await using var cmd = Conn.CreateCommand(c, ""); + var n = 0; + for (var i = 0; i < batch.Length; i++) + { + var (t, v, status, deleted) = batch[i]; + var col = SampleColumnMapper.ToColumns(v); + if (i > 0) sql.Append(", "); + sql.Append('('); + for (var j = 0; j < 9; j++) sql.Append(j == 0 ? "" : ", ").Append('$').Append(++n); + sql.Append(')'); + QuestDbConnection.BindTimestamp(cmd, t); + QuestDbConnection.BindText(cmd, tag); + QuestDbConnection.BindByte(cmd, col.ValueKind); + cmd.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Double, Value = (object?)col.NumValue ?? DBNull.Value }); + cmd.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Bigint, Value = (object?)col.IntValue ?? DBNull.Value }); + QuestDbConnection.BindText(cmd, col.StrValue); + QuestDbConnection.BindLong(cmd, SampleColumnMapper.ToLongStatus(status)); + cmd.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Boolean, Value = deleted }); + QuestDbConnection.BindTimestamp(cmd, Round(t).AddSeconds(1)); + } + cmd.CommandText = sql.ToString(); + await cmd.ExecuteNonQueryAsync(); + } + } + await Wal.WaitAsync(Conn.SampleTable, CancellationToken.None); + } + + /// Runs a statement against this table set (test-only escape hatch for hand-made rows), then waits for WAL visibility of the sample table. + /// The statement; parameters are bound positionally by . + /// Binds the parameters. + public async Task ExecuteAsync(string sql, Action bind) + { + await using (var c = await Conn.OpenAsync(CancellationToken.None)) + { + await using var cmd = Conn.CreateCommand(c, sql); + bind(cmd); + await cmd.ExecuteNonQueryAsync(); + } + await Wal.WaitAsync(Conn.SampleTable, CancellationToken.None); + } + + /// + public async Task> ReadBackAsync(string tag) + { + await using var c = await Conn.OpenAsync(CancellationToken.None); + await using var cmd = Conn.CreateCommand(c, + $"SELECT ts, kind, num_value, int_value, str_value, status FROM {Conn.SampleTable} " + + "WHERE tag = $1 AND NOT deleted ORDER BY ts"); + QuestDbConnection.BindText(cmd, tag); + var list = new List<(DateTime, HistorianValue, uint)>(); + await using var r = await cmd.ExecuteReaderAsync(); + while (await r.ReadAsync()) + { + var v = QuestDbHistorianDataSource.ToValue(QuestDbConnection.ReadKind(r, 1), + r.IsDBNull(2) ? null : r.GetDouble(2), r.IsDBNull(3) ? null : r.GetInt64(3), r.IsDBNull(4) ? null : r.GetString(4)); + // Verbatim: no read-side clamp. A NULL status cannot be written by any code path; it reads as the raw 0 bits. + var status = r.IsDBNull(5) ? 0u : SampleColumnMapper.FromLongStatus(r.GetInt64(5)); + list.Add((QuestDbConnection.ReadUtc(r, 0), v, status)); + } + return list; + } + + /// + public async Task TagExistsAsync(string tag) + { + await using var c = await Conn.OpenAsync(CancellationToken.None); + await using var cmd = Conn.CreateCommand(c, $"SELECT count() FROM {Conn.TagTable} WHERE name = $1"); + QuestDbConnection.BindText(cmd, tag); + return Convert.ToInt64(await cmd.ExecuteScalarAsync()) > 0; + } + + /// + public async Task TagDataTypeAsync(string tag) + { + await using var c = await Conn.OpenAsync(CancellationToken.None); + await using var cmd = Conn.CreateCommand(c, $"SELECT data_type FROM {Conn.TagTable} WHERE name = $1 LIMIT 1"); + QuestDbConnection.BindText(cmd, tag); + await using var r = await cmd.ExecuteReaderAsync(); + return await r.ReadAsync() ? (DriverDataType)QuestDbConnection.ReadKind(r, 0) : null; + } + + /// + public async ValueTask DisposeAsync() + { + await Conn.DisposeAsync(); + await _fx.DropPrefixAsync(_prefix); // early; the fixture also drops every prefix it handed out + } +} diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/ReadTests.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/ReadTests.cs new file mode 100644 index 00000000..d0f29d5a --- /dev/null +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/ReadTests.cs @@ -0,0 +1,266 @@ +using System.Text; +using Npgsql; +using NpgsqlTypes; +using Shouldly; +using Xunit; +using ZB.MOM.WW.OtOpcUa.Core.Abstractions; +using ZB.MOM.WW.OtOpcUa.Driver.Historian.Conformance; +using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common; + +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests; + +/// +/// QuestDB-specific read behaviour (the shared Raw/AtTime/Processed literals run through the conformance suite): +/// tombstones, the pre-1970 / whole-range bucket math, reverse paging at one value per page, QuestDB's LONG NULL +/// sentinel, the kind-decides-the-value rule, AtTime chunking, Events and failure accounting. +/// +[Collection(QuestDbCollection.Name)] +public sealed class ReadTests(QuestDbFixture fx) : IAsyncLifetime +{ + private static readonly CancellationToken Ct = CancellationToken.None; + + private const uint Good = ConformanceData.Good; + private const uint Bad = ConformanceData.Bad; + private const uint GoodCalc = 0x00000401u; + private const uint UncertainCalc = 0x40A40401u; + private const uint UncertainInterpolated = 0x40A40402u; + private const uint GoodInterpolated = 0x00000402u; + private const uint NoDataCalc = 0x809B0401u; + + private QuestDbConformanceBackend _b = null!; + + private static DateTime At(double s) => ConformanceData.At(s); + + public async ValueTask InitializeAsync() => _b = await QuestDbConformanceBackend.CreateConcreteAsync(fx); + + public async ValueTask DisposeAsync() + { + if (_b is not null) await _b.DisposeAsync(); + } + + private static double?[] Values(HistoryReadResult r) => r.Samples.Select(s => (double?)s.Value).ToArray(); + + /// What T6's DeleteAtTime writes: the same (ts, tag), deleted = true, value columns NULL, status BadNoData. + private Task TombstoneAsync(string tag, DateTime t) => + _b.InsertRowsAsync(tag, [(t, HistorianValue.Empty, HistorianStatusCodes.BadNoData, true)]); + + [Fact] + public async Task Tombstoned_rows_are_invisible_to_raw_attime_and_processed() + { + await ConformanceData.SeedAAsync(_b); // +0=10, +10=20, +20=Bad 30, +30=40, +40=50 + await TombstoneAsync("A", At(10)); // DEDUP replaces the +10 row with a tombstone + (await _b.ReadBackAsync("A")).Count.ShouldBe(4); + var src = _b.CreateDataSource(); + + var raw = await src.ReadRawAsync("A", At(0), At(100), 0, Ct); + Values(raw).ShouldBe([10d, 30d, 40d, 50d]); + raw.Samples.Select(s => s.StatusCode).ShouldBe([Good, Bad, Good, Good]); + + // +10 is no longer an exact hit: previous is +0 = 10 (Good), next is the Bad +20 ⇒ held, Uncertain. + var at = (await src.ReadAtTimeAsync("A", [At(10), At(5)], Ct)).Samples; + at.Select(s => s.Value).ShouldBe([(object?)10d, 10d]); + at.ShouldAllBe(s => s.StatusCode == UncertainInterpolated); + + // B0 [0,20) holds only {10}: Good, count 1. A visible tombstone would be a Bad sample ⇒ Uncertain, PercentBad 50. + async Task B0(HistoryAggregateType a) => + (await src.ReadProcessedAsync("A", At(0), At(80), TimeSpan.FromSeconds(20), a, Ct)).Samples[0]; + var avg = await B0(HistoryAggregateType.Average); + avg.Value.ShouldBe(10d); + avg.StatusCode.ShouldBe(GoodCalc); + (await B0(HistoryAggregateType.Count)).Value.ShouldBe(1); + (await B0(HistoryAggregateType.PercentBad)).Value.ShouldBe(0d); + (await B0(HistoryAggregateType.End)).Value.ShouldBe(10d); + } + + [Fact] + public async Task A_whole_range_processed_read_from_1601_to_9999_with_interval_zero_does_not_overflow() + { + await ConformanceData.SeedAAsync(_b); + var src = _b.CreateDataSource(); + var start = new DateTime(1601, 1, 1, 0, 0, 0, DateTimeKind.Utc); + var end = new DateTime(9999, 12, 31, 0, 0, 0, DateTimeKind.Utc); + foreach (var agg in new[] { HistoryAggregateType.Average, HistoryAggregateType.Start, HistoryAggregateType.MaximumActualTime }) + { + var s = (await src.ReadProcessedAsync("A", start, end, TimeSpan.Zero, agg, Ct)).Samples.Single(); + s.StatusCode.ShouldBe(UncertainCalc, agg.ToString()); + } + (await src.ReadProcessedAsync("A", start, end, TimeSpan.Zero, HistoryAggregateType.Average, Ct)).Samples.Single().Value.ShouldBe(30d); + (await src.ReadProcessedAsync("A", start, end, TimeSpan.Zero, HistoryAggregateType.Start, Ct)).Samples.Single().Value.ShouldBe(10d); + var max = (await src.ReadProcessedAsync("A", start, end, TimeSpan.Zero, HistoryAggregateType.MaximumActualTime, Ct)).Samples.Single(); + max.Value.ShouldBe(50d); + max.SourceTimestampUtc.ShouldBe(_b.Round(At(40))); + // The extreme DateTime range: MinValue/MaxValue are nudged (never bound as ±infinity, which would match nothing). + (await src.ReadProcessedAsync("A", DateTime.MinValue, DateTime.MaxValue, TimeSpan.Zero, HistoryAggregateType.Total, Ct)) + .Samples.Single().Value.ShouldBe(120d); + // A pre-1970 window holds no data and is not an error. + (await src.ReadProcessedAsync("A", start, new DateTime(1969, 1, 1, 0, 0, 0, DateTimeKind.Utc), TimeSpan.FromDays(3650), + HistoryAggregateType.Count, Ct)).Samples.ShouldAllBe(x => x.StatusCode == NoDataCalc); + src.GetHealthSnapshot().TotalFailures.ShouldBe(0); + } + + /// + /// The node manager's reverse paging at NumValuesPerNode = 1 (OtOpcUaNodeManager + HistoryPaging): resume at + /// the last timestamp inclusive and trim it; when the trimmed page stalls on that timestamp, point-read it and step + /// one tick down. One tick below a µs-aligned time rounds to the previous µs, so the walk advances. + /// + [Fact] + public async Task A_reverse_paging_walk_with_one_value_per_page_returns_every_row() + { + await ConformanceData.SeedAAsync(_b); + var src = _b.CreateDataSource(); + var end = At(-1); + DateTime start = At(100); + var resumed = false; + var emitted = new List(); + for (var guard = 0; guard < 50; guard++) + { + var page = (await src.ReadRawAsync("A", start, end, 1, Ct)).Samples; + var trimmed = resumed ? page.Where(s => s.SourceTimestampUtc != start).ToList() : page.ToList(); + if (page.Count == 0) break; + if (resumed && (trimmed.Count == 0 || trimmed[^1].SourceTimestampUtc == start)) + { + // Stalled on the boundary: the tie cluster at `start` is one sample (already emitted) ⇒ step down a tick. + (await src.ReadRawAsync("A", start, start, 2, Ct)).Samples.Count.ShouldBe(1); + if (start.AddTicks(-1) <= end) break; + start = start.AddTicks(-1); + resumed = false; + continue; + } + emitted.AddRange(trimmed.Select(s => s.Value)); + start = page[^1].SourceTimestampUtc!.Value; + resumed = true; + } + emitted.ShouldBe([50d, 40d, 30d, 20d, 10d]); + } + + [Fact] + public async Task An_int64_of_long_MinValue_is_served_as_long_MinValue_not_zero() + { + // QuestDB stores long.MinValue in a LONG as NULL (its NULL sentinel); num_value keeps -2^63 exactly (RC2). + await _b.SeedAsync("L", + (At(0), HistorianValue.FromInt64(long.MinValue), Good), + (At(1), HistorianValue.FromInt64(long.MinValue + 1), Good)); + var src = _b.CreateDataSource(); + (await src.ReadRawAsync("L", At(0), At(10), 0, Ct)).Samples.Select(s => s.Value) + .ShouldBe([(object?)long.MinValue, long.MinValue + 1]); + (await src.ReadAtTimeAsync("L", [At(0)], Ct)).Samples.Single().Value.ShouldBe(long.MinValue); + (await src.ReadProcessedAsync("L", At(0), At(10), TimeSpan.Zero, HistoryAggregateType.Start, Ct)) + .Samples.Single().Value.ShouldBe(long.MinValue); + } + + [Fact] + public async Task The_kind_not_the_populated_column_decides_the_value() + { + // An Int64-kind row whose num_value is NULL (a NaN-free double view lost) still serves its int_value, and a + // Double-kind row carrying a stray int_value still serves num_value. + await _b.SeedAsync("K"); + await _b.ExecuteAsync( + $"INSERT INTO {_b.Conn.SampleTable} (ts, tag, kind, num_value, int_value, status, deleted, server_ts) " + + "VALUES ($1, $2, $3, NULL, $4, $5, false, $6), ($7, $2, $8, $9, $10, $5, false, $11)", + cmd => + { + QuestDbConnection.BindTimestamp(cmd, At(0)); // $1 + QuestDbConnection.BindText(cmd, "K"); // $2 + QuestDbConnection.BindByte(cmd, (byte)HistorianValueKind.Int64); // $3 + QuestDbConnection.BindLong(cmd, 7); // $4 + QuestDbConnection.BindLong(cmd, 0); // $5 status Good + QuestDbConnection.BindTimestamp(cmd, At(1)); // $6 + QuestDbConnection.BindTimestamp(cmd, At(2)); // $7 + QuestDbConnection.BindByte(cmd, (byte)HistorianValueKind.Double); // $8 + cmd.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Double, Value = 2.5 }); // $9 + QuestDbConnection.BindLong(cmd, 99); // $10 + QuestDbConnection.BindTimestamp(cmd, At(3)); // $11 + }); + var s = (await _b.CreateDataSource().ReadRawAsync("K", At(0), At(10), 0, Ct)).Samples; + s.Select(x => x.Value).ShouldBe([(object?)7L, 2.5d]); + } + + [Fact] + public async Task AtTime_over_several_statement_chunks_keeps_request_length_and_order() + { + await ConformanceData.SeedAAsync(_b); + // 140 distinct times (3 statements of ≤ 50), requested in descending order, plus a duplicate of the first. + var times = Enumerable.Range(0, 140).Select(i => At((390 - i * 3) / 10.0)).ToList(); + times.Add(times[0]); + var s = (await _b.CreateDataSource().ReadAtTimeAsync("A", times, Ct)).Samples; + s.Count.ShouldBe(times.Count); + s.Select(x => x.SourceTimestampUtc).ShouldBe(times.Select(t => (DateTime?)_b.Round(t))); + s[0].Value.ShouldBe(49d); // +39: 40 + 10 × 9/10 + s[0].StatusCode.ShouldBe(GoodInterpolated); + s[^1].Value.ShouldBe(49d); // the duplicate + s[100].Value.ShouldBe(19d); // +9: 10 + 10 × 9/10, from the third chunk + s[139].Value.ShouldBeNull(); // -2.7 s: no previous sample + s[139].StatusCode.ShouldBe(0x809B0000u); // BadNoData + } + + [Fact] + public async Task Events_are_filtered_by_source_and_window_ascending_and_limited_with_a_continuation() + { + await _b.Boot.EnsureAsync(Ct); + async Task Event(DateTime t, string id, string source, int severity, string? message) + { + await _b.ExecuteAsync( + $"INSERT INTO {_b.Conn.EventTable} (ts, event_id, source, severity, message, received_utc) VALUES ($1, $2, $3, $4, $5, $6)", + cmd => + { + QuestDbConnection.BindTimestamp(cmd, t); + QuestDbConnection.BindText(cmd, id); + QuestDbConnection.BindText(cmd, source); + cmd.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Integer, Value = severity }); + QuestDbConnection.BindText(cmd, message); + QuestDbConnection.BindTimestamp(cmd, t.AddSeconds(2)); + }); + await _b.Wal.WaitAsync(_b.Conn.EventTable, Ct); + } + await Event(At(3), "e3", "src1", 700, "three"); + await Event(At(1), "e1", "src1", 70000, "one"); + await Event(At(2), "e2", "src2", 500, null); + await Event(At(9), "e9", "src1", 500, "out of window"); + + var src = _b.CreateDataSource(); + var all = await src.ReadEventsAsync(null, At(0), At(5), 0, Ct); + all.Events.Select(e => e.EventId).ShouldBe(["e1", "e2", "e3"]); + all.ContinuationPoint.ShouldBeNull(); + all.Events[0].Severity.ShouldBe((ushort)65535); + all.Events[0].EventTimeUtc.ShouldBe(_b.Round(At(1))); + all.Events[0].ReceivedTimeUtc.ShouldBe(_b.Round(At(1)).AddSeconds(2)); + all.Events[1].Message.ShouldBeNull(); + + var one = await src.ReadEventsAsync("src1", At(5), At(0), 1, Ct); // reversed window is swapped + one.Events.Select(e => e.EventId).ShouldBe(["e1"]); + Encoding.UTF8.GetString(one.ContinuationPoint!).ShouldBe(EventContinuation.Encode(_b.Round(At(3)), "e3")); + (await src.ReadEventsAsync("src1", At(0), At(5), 2, Ct)).ContinuationPoint.ShouldBeNull(); + (await src.ReadEventsAsync(new string('x', 401), At(0), At(5), 0, Ct)).Events.ShouldBeEmpty(); + } + + [Fact] + public async Task A_failed_read_is_rethrown_and_recorded_in_health_without_counting_client_errors() + { + // A table set that does not exist, with AutoCreateSchema off: every read that reaches the database fails. + var prefix = fx.UniquePrefix(); + var o = fx.Options(prefix, autoCreate: false); + await using var conn = new QuestDbConnection(o); + var boot = new QuestDbSchemaBootstrapper(conn, o); + using var src = new QuestDbHistorianDataSource(conn, boot, new QuestDbTagCatalog(conn, boot, new QuestDbWalVisibility(conn, o.VisibilityTimeout))); + + await Should.ThrowAsync(() => src.ReadRawAsync("A", At(0), At(100), 0, Ct)); + await Should.ThrowAsync(() => src.ReadEventsAsync(null, At(0), At(5), 0, Ct)); + // Client-request errors are thrown before any I/O and are not backend failures. + await Should.ThrowAsync(() => + src.ReadProcessedAsync("A", At(0), At(5), TimeSpan.Zero, HistoryAggregateType.Interpolative, Ct)); + await Should.ThrowAsync(() => + src.ReadProcessedAsync("A", At(0), At(100), TimeSpan.FromTicks(10), HistoryAggregateType.Count, Ct)); + + var h = src.GetHealthSnapshot(); + h.TotalQueries.ShouldBe(2); + h.TotalFailures.ShouldBe(2); + h.ConsecutiveFailures.ShouldBe(2); + h.LastError.ShouldNotBeNull(); + h.LastError.ShouldStartWith(nameof(HistorianSchemaMissingException)); + h.ProcessConnectionOpen.ShouldBeFalse(); + + // The healthy backend's own data source is unaffected (a failure fails only its call). + await ConformanceData.SeedAAsync(_b); + (await _b.CreateDataSource().ReadRawAsync("A", At(0), At(100), 0, Ct)).Samples.Count.ShouldBe(5); + } +} From 9d90dc94577b38b09abe2d0dd2c63d992a19b650 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Sun, 11 Oct 2026 02:16:14 -0400 Subject: [PATCH 15/23] fix(historian): clustered existence reads + tag-predicate tests (T4 review) Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe --- .../QuestDbHistorianValueWriter.cs | 56 +++++++++++--- .../WriteTests.cs | 75 +++++++++++++++++++ 2 files changed, 122 insertions(+), 9 deletions(-) diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianValueWriter.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianValueWriter.cs index de1d790c..791ef010 100644 --- a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianValueWriter.cs +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianValueWriter.cs @@ -98,19 +98,57 @@ internal sealed class QuestDbHistorianValueWriter : IHistorianValueWriter } } - // T0 Q11 range form: tombstones are NOT filtered out (a deleted row must not be resurrected). + /// A gap between consecutive batch times larger than this starts a new existence-query range. + internal static readonly TimeSpan ExistenceClusterGap = TimeSpan.FromMinutes(1); + + /// + /// Splits ascending into inclusive [Lo, Hi] ranges: a new range starts where the gap to the + /// previous time exceeds or the range already holds times. One outlier + /// time therefore costs its own tiny query instead of widening one range over every stored row in between. + /// + /// Times in ascending order. + /// Largest gap kept inside one range. + /// Most times in one range (>= 1). + /// The ranges, ascending. + internal static IReadOnlyList<(DateTime Lo, DateTime Hi)> Cluster(IReadOnlyList sortedTimes, TimeSpan gap, int maxPerCluster) + { + ArgumentOutOfRangeException.ThrowIfLessThan(maxPerCluster, 1); + var result = new List<(DateTime, DateTime)>(); + if (sortedTimes.Count == 0) return result; + var lo = sortedTimes[0]; + var prev = lo; + var n = 1; + for (var i = 1; i < sortedTimes.Count; i++) + { + var t = sortedTimes[i]; + if (t - prev > gap || n >= maxPerCluster) + { + result.Add((lo, prev)); + lo = t; + n = 0; + } + prev = t; + n++; + } + result.Add((lo, prev)); + return result; + } + + // T0 Q11 range form, one query per cluster: tombstones are NOT filtered out (a deleted row must not be resurrected). private async Task> ExistingTicksAsync(string tag, List rows, CancellationToken ct) { - var min = rows.Min(r => QuestDbConnection.Round(r.TimeUtc)); - var max = rows.Max(r => QuestDbConnection.Round(r.TimeUtc)); + var times = rows.Select(r => QuestDbConnection.Round(r.TimeUtc)).OrderBy(t => t).ToList(); var set = new HashSet(); await using var conn = await _conn.OpenAsync(ct).ConfigureAwait(false); - await using var cmd = _conn.CreateCommand(conn, $"SELECT ts FROM {_conn.SampleTable} WHERE tag = $1 AND ts >= $2 AND ts <= $3"); - QuestDbConnection.BindText(cmd, tag); - QuestDbConnection.BindTimestamp(cmd, min); - QuestDbConnection.BindTimestamp(cmd, max); - await using var r = await cmd.ExecuteReaderAsync(ct).ConfigureAwait(false); - while (await r.ReadAsync(ct).ConfigureAwait(false)) set.Add(QuestDbConnection.ReadUtc(r, 0).Ticks); + foreach (var (lo, hi) in Cluster(times, ExistenceClusterGap, BatchRows)) + { + await using var cmd = _conn.CreateCommand(conn, $"SELECT ts FROM {_conn.SampleTable} WHERE tag = $1 AND ts >= $2 AND ts <= $3"); + QuestDbConnection.BindText(cmd, tag); + QuestDbConnection.BindTimestamp(cmd, lo); + QuestDbConnection.BindTimestamp(cmd, hi); + await using var r = await cmd.ExecuteReaderAsync(ct).ConfigureAwait(false); + while (await r.ReadAsync(ct).ConfigureAwait(false)) set.Add(QuestDbConnection.ReadUtc(r, 0).Ticks); + } return set; } diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/WriteTests.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/WriteTests.cs index 3f8a40f7..39e75e1b 100644 --- a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/WriteTests.cs +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/WriteTests.cs @@ -218,4 +218,79 @@ public sealed class WriteTests(QuestDbFixture fx) cts.Cancel(); (await w.WriteLiveValuesAsync("x", [new(T0, HistorianValue.FromDouble(1), 0)], cts.Token)).ShouldBeFalse(); } + + private static async Task SeedRowAsync(QuestDbConnection c, QuestDbWalVisibility wal, string tag, DateTime t, double v) + { + await using (var conn = await c.OpenAsync(default)) + { + await using var cmd = c.CreateCommand(conn, + $"INSERT INTO {c.SampleTable} (ts, tag, kind, num_value, int_value, str_value, status, deleted, server_ts) VALUES ($1,$2,1,$3,NULL,NULL,0,false,$4)"); + QuestDbConnection.BindTimestamp(cmd, t); + QuestDbConnection.BindText(cmd, tag); + cmd.Parameters.Add(new Npgsql.NpgsqlParameter { NpgsqlDbType = NpgsqlTypes.NpgsqlDbType.Double, Value = v }); + QuestDbConnection.BindTimestamp(cmd, t); + await cmd.ExecuteNonQueryAsync(); + } + await wal.WaitAsync(c.SampleTable, default); + } + + [Fact] + public async Task A_row_on_another_tag_at_the_same_timestamp_does_not_suppress_the_write() + { + fx.SkipIfUnavailable(); + var (w, c) = Make(); + var wal = new QuestDbWalVisibility(c, TimeSpan.FromSeconds(5)); + (await w.WriteLiveValuesAsync("tag.A", [new(T0, HistorianValue.FromDouble(1), 0)], CancellationToken.None)).ShouldBeTrue(); + (await w.WriteLiveValuesAsync("tag.B", [new(T0, HistorianValue.FromDouble(2), 0)], CancellationToken.None)).ShouldBeTrue(); + (await RowsAsync(c, "tag.A")).Single().N.ShouldBe(1d); + (await RowsAsync(c, "tag.B")).Single().N.ShouldBe(2d); + } + + [Fact] + public async Task An_absent_interior_timestamp_is_written_while_both_present_ends_are_left_alone() + { + fx.SkipIfUnavailable(); + var (w, c) = Make(); + var wal = new QuestDbWalVisibility(c, TimeSpan.FromSeconds(5)); + await w.WriteLiveValuesAsync("tag.ends", [new(T0.AddSeconds(100), HistorianValue.FromDouble(0), 0)], CancellationToken.None); // creates the tag/tables + await SeedRowAsync(c, wal, "tag.ends", T0, 10); + await SeedRowAsync(c, wal, "tag.ends", T0.AddSeconds(20), 30); + (await w.WriteLiveValuesAsync("tag.ends", + [new(T0, HistorianValue.FromDouble(-1), 0), new(T0.AddSeconds(10), HistorianValue.FromDouble(20), 0), new(T0.AddSeconds(20), HistorianValue.FromDouble(-3), 0)], + CancellationToken.None)).ShouldBeTrue(); + var rows = (await RowsAsync(c, "tag.ends")).Where(r => r.T <= T0.AddSeconds(20)).ToList(); + rows.Select(r => r.N).ShouldBe([10d, 20d, 30d]); + } + + [Fact] + public async Task An_outlier_timestamp_does_not_widen_the_existence_read_over_the_stored_rows_between() + { + fx.SkipIfUnavailable(); + var (w, c) = Make(); + // Behavioural check of the whole path with a far-apart batch: both samples land, a redelivery adds nothing. + var far = T0.AddYears(1); + HistorizationValue[] vals = [new(T0, HistorianValue.FromDouble(1), 0), new(far, HistorianValue.FromDouble(2), 0)]; + (await w.WriteLiveValuesAsync("tag.far", vals, CancellationToken.None)).ShouldBeTrue(); + (await w.WriteLiveValuesAsync("tag.far", vals, CancellationToken.None)).ShouldBeTrue(); + (await RowsAsync(c, "tag.far")).Count.ShouldBe(2); + } + + [Fact] + public void Cluster_splits_on_a_gap_over_the_limit_and_at_the_per_cluster_cap() + { + var g = TimeSpan.FromMinutes(1); + QuestDbHistorianValueWriter.Cluster([], g, 10).ShouldBeEmpty(); + QuestDbHistorianValueWriter.Cluster([T0], g, 10).ShouldBe([(T0, T0)]); + // exactly the gap stays together; one tick over splits + QuestDbHistorianValueWriter.Cluster([T0, T0 + g], g, 10).ShouldBe([(T0, T0 + g)]); + QuestDbHistorianValueWriter.Cluster([T0, T0 + g + TimeSpan.FromTicks(10)], g, 10) + .ShouldBe([(T0, T0), (T0 + g + TimeSpan.FromTicks(10), T0 + g + TimeSpan.FromTicks(10))]); + // outlier gets its own range + var far = T0.AddYears(1); + QuestDbHistorianValueWriter.Cluster([T0, T0.AddSeconds(1), far], g, 10).ShouldBe([(T0, T0.AddSeconds(1)), (far, far)]); + // cap: 5 close times, max 2 per cluster + var five = Enumerable.Range(0, 5).Select(i => T0.AddSeconds(i)).ToList(); + QuestDbHistorianValueWriter.Cluster(five, g, 2) + .ShouldBe([(five[0], five[1]), (five[2], five[3]), (five[4], five[4])]); + } } From 13d628a07f47cddecfacff96abad19c427d524e7 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Sun, 11 Oct 2026 02:23:21 -0400 Subject: [PATCH 16/23] feat(historian): QuestDb provider, registry + consumer-gated validator, compose env Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe --- docker-dev/docker-compose.yml | 6 + .../QuestDbHistorianProvider.cs | 106 ++++++++++++++++++ .../Configuration/HistorianBackendRegistry.cs | 13 +++ .../ServerHistorianOptionsValidator.cs | 7 ++ .../ZB.MOM.WW.OtOpcUa.Host.csproj | 1 + .../QuestDbHistorianProviderTests.cs | 90 +++++++++++++++ .../ServerHistorianOptionsValidatorTests.cs | 85 ++++++++++++++ .../HistorianBackendRegistryTests.cs | 43 +++++++ 8 files changed, 351 insertions(+) create mode 100644 src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianProvider.cs create mode 100644 tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests/QuestDbHistorianProviderTests.cs diff --git a/docker-dev/docker-compose.yml b/docker-dev/docker-compose.yml index 3d410597..d6e439b8 100644 --- a/docker-dev/docker-compose.yml +++ b/docker-dev/docker-compose.yml @@ -441,6 +441,9 @@ services: # docker compose -f docker-dev/docker-compose.yml up -d central-1 central-2 ServerHistorian__Backend: "${OTOPCUA_HISTORIAN_BACKEND:-Gateway}" ServerHistorian__SqlServer__ConnectionString: "${OTOPCUA_HISTORIAN_SQL:-Server=sql,1433;Database=OtOpcUaHistorian;User Id=sa;Password=OtOpcUa!Dev123;TrustServerCertificate=True;Encrypt=False}" + # QuestDb = the QuestDB historian; the `questdb` service comes from the docker-dev/docker-compose.questdb.yml overlay + # (add `-f docker-dev/docker-compose.questdb.yml` and OTOPCUA_HISTORIAN_BACKEND=QuestDb). + ServerHistorian__QuestDb__ConnectionString: "${OTOPCUA_HISTORIAN_QUESTDB:-Host=questdb;Port=8812;Username=admin;Password=quest;Database=qdb;ServerCompatibilityMode=NoTypeLoading}" # Continuous historization of driver tag values (Gitea #491's live gate). Gated on BOTH # ContinuousHistorization:Enabled AND ServerHistorian:Enabled — with either off no recorder is # spawned, so this stays inert on the default rig. OutboxPath is the FasterLog crash boundary and @@ -574,6 +577,9 @@ services: # docker compose -f docker-dev/docker-compose.yml up -d central-1 central-2 ServerHistorian__Backend: "${OTOPCUA_HISTORIAN_BACKEND:-Gateway}" ServerHistorian__SqlServer__ConnectionString: "${OTOPCUA_HISTORIAN_SQL:-Server=sql,1433;Database=OtOpcUaHistorian;User Id=sa;Password=OtOpcUa!Dev123;TrustServerCertificate=True;Encrypt=False}" + # QuestDb = the QuestDB historian; the `questdb` service comes from the docker-dev/docker-compose.questdb.yml overlay + # (add `-f docker-dev/docker-compose.questdb.yml` and OTOPCUA_HISTORIAN_BACKEND=QuestDb). + ServerHistorian__QuestDb__ConnectionString: "${OTOPCUA_HISTORIAN_QUESTDB:-Host=questdb;Port=8812;Username=admin;Password=quest;Database=qdb;ServerCompatibilityMode=NoTypeLoading}" # Continuous historization — see central-1 for the full note + the gate recipe. Both centrals # carry it so the pair is symmetric; :9200 round-robins them, and the recorder must exist on # whichever node holds the drivers. diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianProvider.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianProvider.cs new file mode 100644 index 00000000..39af8413 --- /dev/null +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianProvider.cs @@ -0,0 +1,106 @@ +using System.Collections.Frozen; +using Microsoft.Extensions.Logging; +using Microsoft.Extensions.Logging.Abstractions; +using ZB.MOM.WW.OtOpcUa.Core.Abstractions; +using ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian; +using ZB.MOM.WW.OtOpcUa.Runtime.Historian; + +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb; + +/// +/// for the QuestDB historian. Supports every +/// and HistoryUpdate (soft delete). All seams share ONE connection helper, ONE +/// schema bootstrapper, ONE tag catalog (R7: the only "known tag" authority) and ONE WAL visibility wait, so the +/// schema is ensured once and known tags are cached once. +/// +/// +/// The shared state is built on the first Create* call, not in the constructor: the constructor does no +/// validation and no I/O, so resolving the provider for a backend nobody consumes never fails on its options +/// (they are validated, consumer-gated, by the host's ServerHistorianOptionsValidator). The first +/// Create* call's service provider supplies the logger factory the shared pieces log through. +/// +public sealed class QuestDbHistorianProvider : IHistorianBackendProvider +{ + private static readonly FrozenSet Kinds = + Enum.GetValues().ToFrozenSet(); + + private readonly QuestDbHistorianOptions _options; + private readonly object _initLock = new(); + private Shared? _shared; + + /// Creates the provider over the ServerHistorian:QuestDb options (validated elsewhere). + /// The ServerHistorian:QuestDb options. + public QuestDbHistorianProvider(QuestDbHistorianOptions options) + { + _options = options ?? throw new ArgumentNullException(nameof(options)); + } + + /// + public string Name => "QuestDb"; + + /// + public IReadOnlySet SupportedKinds => Kinds; + + private static ILoggerFactory Loggers(IServiceProvider sp) => + (sp.GetService(typeof(ILoggerFactory)) as ILoggerFactory) ?? NullLoggerFactory.Instance; + + private Shared Get(IServiceProvider sp) + { + if (Volatile.Read(ref _shared) is { } built) return built; + lock (_initLock) + { + if (_shared is { } raced) return raced; + var loggers = Loggers(sp); + var conn = new QuestDbConnection(_options); + var bootstrap = new QuestDbSchemaBootstrapper(conn, _options, loggers.CreateLogger()); + var wal = new QuestDbWalVisibility(conn, _options.VisibilityTimeout, loggers.CreateLogger()); + var shared = new Shared(conn, bootstrap, new QuestDbTagCatalog(conn, bootstrap, wal), wal); + Volatile.Write(ref _shared, shared); + return shared; + } + } + + /// + public IHistorianDataSource CreateDataSource(IServiceProvider sp) + { + var s = Get(sp); + return new QuestDbHistorianDataSource(s.Conn, s.Bootstrap, s.Catalog, + Loggers(sp).CreateLogger()); + } + + /// + public IHistorianValueWriter CreateValueWriter(IServiceProvider sp) + { + var s = Get(sp); + return new QuestDbHistorianValueWriter(s.Conn, s.Catalog, s.Wal, + Loggers(sp).CreateLogger()); + } + + /// + public IAlarmHistorianWriter CreateAlarmWriter(IServiceProvider sp) + { + var s = Get(sp); + return new QuestDbAlarmHistorianWriter(s.Conn, s.Bootstrap, s.Wal, + Loggers(sp).CreateLogger()); + } + + /// + public IHistorianProvisioning CreateProvisioning(IServiceProvider sp) + { + var s = Get(sp); + return new QuestDbTagProvisioner(s.Conn, s.Bootstrap, s.Catalog, s.Wal, + Loggers(sp).CreateLogger()); + } + + /// + /// The soft-delete updater: deletes write tombstones (INSERT only, never QuestDB UPDATE/DELETE). + public IHistorianDataUpdater? CreateUpdater(IServiceProvider sp) + { + var s = Get(sp); + return new QuestDbHistorianDataUpdater(s.Conn, s.Catalog, s.Wal, + Loggers(sp).CreateLogger()); + } + + private sealed record Shared( + QuestDbConnection Conn, QuestDbSchemaBootstrapper Bootstrap, QuestDbTagCatalog Catalog, QuestDbWalVisibility Wal); +} diff --git a/src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/HistorianBackendRegistry.cs b/src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/HistorianBackendRegistry.cs index ce68db91..a14e5bd6 100644 --- a/src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/HistorianBackendRegistry.cs +++ b/src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/HistorianBackendRegistry.cs @@ -1,6 +1,7 @@ using Microsoft.Extensions.Configuration; using Microsoft.Extensions.Options; using ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway; +using ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb; using ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer; using ZB.MOM.WW.OtOpcUa.Runtime.Historian; @@ -25,11 +26,23 @@ public static class HistorianBackendRegistry return o; } + /// Backend name for the QuestDB historian. + public const string QuestDbBackendName = "QuestDb"; + + /// Binds the ServerHistorian:QuestDb section (shared by the registry and the validator). + public static QuestDbHistorianOptions BindQuestDb(IConfiguration configuration) + { + var o = new QuestDbHistorianOptions(); + configuration.GetSection($"{ServerHistorianOptions.SectionName}:QuestDb").Bind(o); + return o; + } + private static readonly IReadOnlyDictionary> Factories = new Dictionary>(StringComparer.OrdinalIgnoreCase) { [ServerHistorianOptions.GatewayBackendName] = (o, _) => new GatewayHistorianProvider(o), [SqlServerBackendName] = (_, c) => new SqlServerHistorianProvider(BindSqlServer(c)), + [QuestDbBackendName] = (_, c) => new QuestDbHistorianProvider(BindQuestDb(c)), }; /// The registered backend names, for validation messages. diff --git a/src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/ServerHistorianOptionsValidator.cs b/src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/ServerHistorianOptionsValidator.cs index e421747f..ac51be90 100644 --- a/src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/ServerHistorianOptionsValidator.cs +++ b/src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/ServerHistorianOptionsValidator.cs @@ -78,6 +78,13 @@ public sealed class ServerHistorianOptionsValidator : OptionsValidatorBase + diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests/QuestDbHistorianProviderTests.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests/QuestDbHistorianProviderTests.cs new file mode 100644 index 00000000..b5df992b --- /dev/null +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests/QuestDbHistorianProviderTests.cs @@ -0,0 +1,90 @@ +using Shouldly; +using Xunit; +using ZB.MOM.WW.OtOpcUa.Core.Abstractions; + +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests; + +public sealed class QuestDbHistorianProviderTests +{ + private static QuestDbHistorianProvider Make() => new(new QuestDbHistorianOptions + { + ConnectionString = "Host=localhost;Port=18812;Username=admin;Password=quest;Database=qdb;ServerCompatibilityMode=NoTypeLoading", + }); + + [Fact] + public void Name_and_all_six_kinds() + { + var p = Make(); + p.Name.ShouldBe("QuestDb"); + p.SupportedKinds.ShouldBe(Enum.GetValues(), ignoreOrder: true); + p.SupportedKinds.Count.ShouldBe(6); + } + + [Fact] + public void Creates_all_five_seams_without_touching_questdb() + { + var sp = new NullSp(); + var p = Make(); + p.CreateDataSource(sp).ShouldBeOfType(); + p.CreateValueWriter(sp).ShouldBeOfType(); + p.CreateAlarmWriter(sp).ShouldBeOfType(); + p.CreateProvisioning(sp).ShouldBeOfType(); + p.CreateUpdater(sp).ShouldBeOfType(); + } + + [Fact] + public void Supported_kinds_are_a_frozen_set() => + Make().SupportedKinds.ShouldBeAssignableTo>(); + + /// + /// An invalid table prefix must not fail provider construction (the host resolves the provider even when no + /// consumer is enabled); it surfaces only when a seam is actually built. + /// + [Fact] + public void An_invalid_prefix_fails_only_when_a_seam_is_built() + { + var p = new QuestDbHistorianProvider(new QuestDbHistorianOptions { ConnectionString = "Host=x", TablePrefix = "bad;prefix" }); + p.Name.ShouldBe("QuestDb"); + Should.Throw(() => p.CreateDataSource(new NullSp())); + } + + [Fact] + public void Every_seam_shares_one_connection_bootstrapper_catalog_and_wal_wait() + { + var sp = new NullSp(); + var p = Make(); + var a = (QuestDbHistorianDataSource)p.CreateDataSource(sp); + var b = (QuestDbHistorianDataSource)p.CreateDataSource(sp); + var w = p.CreateValueWriter(sp); + var u = p.CreateUpdater(sp)!; + var prov = p.CreateProvisioning(sp); + var alarm = p.CreateAlarmWriter(sp); + + Field(a, "_conn").ShouldBeSameAs(Field(b, "_conn")); + Field(a, "_conn").ShouldBeSameAs(Field(w, "_conn")); + Field(a, "_conn").ShouldBeSameAs(Field(u, "_conn")); + Field(a, "_conn").ShouldBeSameAs(Field(alarm, "_conn")); + Field(a, "_bootstrap").ShouldBeSameAs(Field(b, "_bootstrap")); + Field(a, "_bootstrap").ShouldBeSameAs(Field(prov, "_bootstrap")); + Field(a, "_bootstrap").ShouldBeSameAs(Field(alarm, "_bootstrap")); + Field(a, "_catalog").ShouldBeSameAs(Field(b, "_catalog")); + Field(a, "_catalog").ShouldBeSameAs(Field(w, "_catalog")); + Field(a, "_catalog").ShouldBeSameAs(Field(u, "_tags")); + Field(a, "_catalog").ShouldBeSameAs(Field(prov, "_catalog")); + Field(w, "_wal").ShouldBeSameAs(Field(u, "_wal")); + Field(w, "_wal").ShouldBeSameAs(Field(prov, "_wal")); + Field(w, "_wal").ShouldBeSameAs(Field(alarm, "_wal")); + } + + [Fact] + public void The_constructor_rejects_null_options() => + Should.Throw(() => new QuestDbHistorianProvider(null!)); + + private static object? Field(object o, string name) => + o.GetType().GetField(name, System.Reflection.BindingFlags.Instance | System.Reflection.BindingFlags.NonPublic)!.GetValue(o); + + private sealed class NullSp : IServiceProvider + { + public object? GetService(Type serviceType) => null; + } +} diff --git a/tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/ServerHistorianOptionsValidatorTests.cs b/tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/ServerHistorianOptionsValidatorTests.cs index 00945b11..9b378e4a 100644 --- a/tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/ServerHistorianOptionsValidatorTests.cs +++ b/tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/ServerHistorianOptionsValidatorTests.cs @@ -375,4 +375,89 @@ public sealed class ServerHistorianOptionsValidatorTests result.Failed.ShouldBeTrue(); result.Failures.ShouldContain(f => f.Contains("ServerHistorian:SqlServer:Schema")); } + + private static ServerHistorianOptionsValidator QuestDbSut(string? connectionString, string? prefix = null, bool alarmHistorianEnabled = false) + { + var values = new Dictionary { ["AlarmHistorian:Enabled"] = alarmHistorianEnabled ? "true" : "false" }; + if (connectionString is not null) values["ServerHistorian:QuestDb:ConnectionString"] = connectionString; + if (prefix is not null) values["ServerHistorian:QuestDb:TablePrefix"] = prefix; + return new ServerHistorianOptionsValidator(new ConfigurationBuilder().AddInMemoryCollection(values).Build()); + } + + /// QuestDb with a connection string passes; the gateway Endpoint/ApiKey/UseTls checks do not fire. + [Fact] + public void QuestDb_backend_with_connection_string_passes_without_gateway_settings() + { + var result = QuestDbSut("Host=questdb;Port=8812").Validate(null, + new ServerHistorianOptions { Backend = "QuestDb", Enabled = true }); + + result.Failed.ShouldBeFalse(string.Join("; ", result.Failures ?? [])); + } + + /// QuestDb with no connection string fails, naming the key, and never asks for gateway settings. + [Fact] + public void QuestDb_backend_without_connection_string_fails_naming_the_key() + { + var result = QuestDbSut(null).Validate(null, new ServerHistorianOptions { Backend = "questdb", Enabled = true }); + + result.Failed.ShouldBeTrue(); + result.Failures.ShouldContain(f => f.Contains("ServerHistorian:QuestDb:ConnectionString")); + result.Failures.ShouldNotContain(f => f.Contains("ServerHistorian:Endpoint")); + result.Failures.ShouldNotContain(f => f.Contains("ServerHistorian:ApiKey")); + } + + /// Every QuestDbHistorianOptions.Validate() error is reported, each naming its key. + [Fact] + public void QuestDb_backend_reports_every_options_error_naming_the_key() + { + var configuration = new ConfigurationBuilder().AddInMemoryCollection(new Dictionary + { + ["AlarmHistorian:Enabled"] = "false", + ["ServerHistorian:QuestDb:TablePrefix"] = "bad;prefix", + ["ServerHistorian:QuestDb:RetentionDays"] = "0", + ["ServerHistorian:QuestDb:CommandTimeout"] = "00:00:00", + ["ServerHistorian:QuestDb:VisibilityTimeout"] = "-00:00:01", + }).Build(); + var result = new ServerHistorianOptionsValidator(configuration) + .Validate(null, new ServerHistorianOptions { Backend = "QuestDb", Enabled = true }); + + result.Failed.ShouldBeTrue(); + foreach (var key in new[] { "ConnectionString", "TablePrefix", "RetentionDays", "CommandTimeout", "VisibilityTimeout" }) + result.Failures.ShouldContain(f => f.Contains($"ServerHistorian:QuestDb:{key}"), key); + } + + /// Unconsumed QuestDb section (nothing enabled) is not validated. + [Fact] + public void QuestDb_backend_not_consumed_passes_without_connection_string() => + QuestDbSut(null).Validate(null, new ServerHistorianOptions { Backend = "QuestDb" }).Failed.ShouldBeFalse(); + + /// + /// An invalid ServerHistorian:QuestDb:TablePrefix must not fail the host when nothing consumes the + /// section: does not throw and the validator does not check it. + /// + [Fact] + public void QuestDb_invalid_prefix_not_consumed_neither_resolve_nor_validation_fails() + { + var configuration = new ConfigurationBuilder().AddInMemoryCollection(new Dictionary + { + ["AlarmHistorian:Enabled"] = "false", + ["ServerHistorian:QuestDb:ConnectionString"] = "Host=questdb;Port=8812", + ["ServerHistorian:QuestDb:TablePrefix"] = "bad;prefix", + }).Build(); + var options = new ServerHistorianOptions { Backend = "QuestDb" }; + + Should.NotThrow(() => HistorianBackendRegistry.Resolve(options, configuration)); + new ServerHistorianOptionsValidator(configuration).Validate(null, options).Failed.ShouldBeFalse(); + } + + /// Consumed (here by AlarmHistorian alone), the same invalid prefix is a named validator error. + [Fact] + public void QuestDb_invalid_prefix_consumed_is_a_validator_error_naming_the_key() + { + var options = new ServerHistorianOptions { Backend = "QuestDb" }; + var result = QuestDbSut("Host=questdb;Port=8812", "bad;prefix", alarmHistorianEnabled: true).Validate(null, options); + + result.Failed.ShouldBeTrue(); + result.Failures.ShouldContain(f => f.Contains("ServerHistorian:QuestDb:TablePrefix")); + } } diff --git a/tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests/Configuration/HistorianBackendRegistryTests.cs b/tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests/Configuration/HistorianBackendRegistryTests.cs index 9251df6a..fe66467c 100644 --- a/tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests/Configuration/HistorianBackendRegistryTests.cs +++ b/tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests/Configuration/HistorianBackendRegistryTests.cs @@ -6,6 +6,7 @@ using Xunit; using ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway; using ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.Recorder; using ZB.MOM.WW.OtOpcUa.Core.Abstractions; +using ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb; using ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer; using ZB.MOM.WW.OtOpcUa.Host.Configuration; using ZB.MOM.WW.OtOpcUa.Runtime.Historian; @@ -86,4 +87,46 @@ public sealed class HistorianBackendRegistryTests holder.Updater.ShouldNotBeNull(); holder.Updater!.GetType().Name.ShouldBe("SqlServerHistorianDataUpdater"); } + + private static IConfiguration QuestDbCfg() => new ConfigurationBuilder().AddInMemoryCollection(new Dictionary + { + ["ServerHistorian:QuestDb:ConnectionString"] = "Host=localhost;Port=18812;Username=admin;Password=quest;Database=qdb", + ["ServerHistorian:QuestDb:TablePrefix"] = "rig_", + }).Build(); + + [Theory] + [InlineData("questdb")] + [InlineData("QUESTDB")] + [InlineData("QuestDb")] + public void QuestDb_backend_resolves_case_insensitively(string name) => + HistorianBackendRegistry.Resolve(new ServerHistorianOptions { Backend = name, Enabled = true }, QuestDbCfg()) + .ShouldBeOfType().Name.ShouldBe("QuestDb"); + + [Fact] + public void QuestDb_backend_resolves_and_builds_the_questdb_seams() + { + var sp = new ServiceCollection().BuildServiceProvider(); + var p = HistorianBackendRegistry.Resolve(new ServerHistorianOptions { Backend = "questdb", Enabled = true }, QuestDbCfg()); + + p.ShouldBeOfType(); + HistorianBackendRegistry.KnownBackends.ShouldContain("QuestDb"); + p.SupportedKinds.Count.ShouldBe(6); + p.SupportedKinds.ShouldContain(HistorianValueKind.String); + p.SupportedKinds.ShouldContain(HistorianValueKind.Empty); + p.CreateDataSource(sp).ShouldBeOfType(); + p.CreateValueWriter(sp).GetType().Name.ShouldBe("QuestDbHistorianValueWriter"); + p.CreateAlarmWriter(sp).GetType().Name.ShouldBe("QuestDbAlarmHistorianWriter"); + p.CreateProvisioning(sp).GetType().Name.ShouldBe("QuestDbTagProvisioner"); + var holder = new HistorianUpdaterHolder(p.CreateUpdater(sp)); + holder.Updater.ShouldNotBeNull(); + holder.Updater!.GetType().Name.ShouldBe("QuestDbHistorianDataUpdater"); + } + + [Fact] + public void BindQuestDb_reads_the_QuestDb_section() + { + var o = HistorianBackendRegistry.BindQuestDb(QuestDbCfg()); + o.ConnectionString.ShouldContain("Port=18812"); + o.TablePrefix.ShouldBe("rig_"); + } } From d2be213e865d48bd79b52d8c4e00eab6a1aefe1b Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Sun, 11 Oct 2026 02:30:25 -0400 Subject: [PATCH 17/23] fix(historian): page by the backend's timestamp resolution; QuestDB AtTime stamps the requested time IHistorianDataSource.TimestampResolutionTicks (default 1; QuestDB 10). A drained tie cluster now steps off T by the resolution, so a forward one-value-per-page read on a microsecond backend no longer re-reads the same sample forever (C1). QuestDB AtTime results carry the client's original request time (M3); regression for the NEXT seek skipping a tombstone (M1). Conformance backend wires the real writer and updater. Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe --- .../Historian/IHistorianDataSource.cs | 15 ++++ .../QuestDbHistorianDataSource.cs | 32 +++++-- .../HistoryPaging.cs | 19 ++-- .../OtOpcUaNodeManager.cs | 2 +- .../AtTimeRegressionTests.cs | 85 ++++++++++++++++++ .../QuestDbConformance.cs | 55 ++++++++++++ .../QuestDbConformanceBackend.cs | 9 +- .../HistoryPagingTests.cs | 45 ++++++++++ .../NodeManagerHistoryReadPagingTests.cs | 87 +++++++++++++++++++ 9 files changed, 330 insertions(+), 19 deletions(-) create mode 100644 tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/AtTimeRegressionTests.cs create mode 100644 tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/QuestDbConformance.cs diff --git a/src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/Historian/IHistorianDataSource.cs b/src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/Historian/IHistorianDataSource.cs index 9942954e..beb0dea2 100644 --- a/src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/Historian/IHistorianDataSource.cs +++ b/src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/Historian/IHistorianDataSource.cs @@ -107,4 +107,19 @@ public interface IHistorianDataSource : IDisposable /// /// The current health snapshot for the historian backend. HistorianHealthSnapshot GetHealthSnapshot(); + + /// + /// The backend's storage timestamp resolution, in ticks (100 ns): the smallest step + /// between two distinct stored timestamps. 1 (the default) means full tick precision; QuestDB stores + /// microseconds and reports 10. + /// + /// + /// The server's HistoryRead-Raw paging steps OFF a drained timestamp by this amount (forward + /// T + resolution, reverse T − resolution). Stepping by less than the resolution is not + /// progress on a backend that truncates its inputs: T + 1 tick is truncated back to T, the + /// same sample comes back, and a one-value-per-page forward read loops forever. Stepping by the resolution is + /// still lossless, because no stored timestamp can lie strictly between T and T ± resolution. + /// Same-timestamp ties are unaffected (they are paged within the timestamp, never stepped over). + /// + long TimestampResolutionTicks => 1; } diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianDataSource.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianDataSource.cs index f4b97539..b1d92d01 100644 --- a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianDataSource.cs +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianDataSource.cs @@ -19,8 +19,10 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb; /// /// Timestamps (R6). Every inbound time is normalised with and every /// is bound only through . Read bounds are not -/// clamped to the epoch (a pre-1970 bound is valid); only MinValue/MaxValue are nudged. Results -/// carry the rounded times, so an AtTime result is stamped at the rounded request time. +/// clamped to the epoch (a pre-1970 bound is valid); only MinValue/MaxValue are nudged. Raw and +/// Processed results carry stored (microsecond) times. An AtTime result is evaluated at the rounded request time +/// but stamped with the client's ORIGINAL requested time (as the SQL Server backend does), so a client can match +/// each answer to its request. /// /// /// Reads never create tags (R7). A tag resolves only through ; @@ -44,6 +46,10 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb; /// public sealed class QuestDbHistorianDataSource : IHistorianDataSource { + /// + /// QuestDB stores microseconds: 10 ticks. HistoryRead paging steps off a drained timestamp by this much. + public long TimestampResolutionTicks => 10; + /// Most AtTime request times per statement (two UNION ALL branches each; QuestDB 10.0.1 refuses more than 126 branches). public const int AtTimeChunkSize = 50; @@ -139,8 +145,9 @@ public sealed class QuestDbHistorianDataSource : IHistorianDataSource /// LIMIT 1 seeks, UNION ALL-ed, at most times per statement (spec §3, /// T0 Q9). then decides each value exactly as for SQL Server (exact hit, /// no previous ⇒ BadNoData, inside an outage, held before an outage, linear, stepped, after now). The - /// result matches the request in length and order (duplicates included), each entry stamped at its rounded - /// time. Unknown tag ⇒ every entry BadNoData. + /// result matches the request in length and order (duplicates included). Each entry is evaluated at its rounded + /// time but stamped with the requested time itself (UTC-normalised as does, + /// not truncated), matching the SQL Server backend. Unknown tag ⇒ every entry BadNoData. /// public async Task ReadAtTimeAsync( string fullReference, IReadOnlyList timestampsUtc, CancellationToken cancellationToken) @@ -197,10 +204,11 @@ public sealed class QuestDbHistorianDataSource : IHistorianDataSource var result = new DataValueSnapshot[rounded.Length]; for (var i = 0; i < result.Length; i++) { - var t = rounded[i]; - result[i] = byTicks.TryGetValue(t.Ticks, out var v) - ? v - : new DataValueSnapshot(null, HistorianStatusCodes.BadNoData, t, now); + // Seek/evaluate at the rounded time; stamp the client's own requested time (M3). + var requested = AsUtc(timestampsUtc[i]); + result[i] = byTicks.TryGetValue(rounded[i].Ticks, out var v) + ? v with { SourceTimestampUtc = requested } + : new DataValueSnapshot(null, HistorianStatusCodes.BadNoData, requested, now); } RecordSuccess(); @@ -213,6 +221,14 @@ public sealed class QuestDbHistorianDataSource : IHistorianDataSource } } + /// The UTC normalisation of without its truncation or nudging. + private static DateTime AsUtc(DateTime t) => t.Kind switch + { + DateTimeKind.Local => t.ToUniversalTime(), + DateTimeKind.Utc => t, + _ => DateTime.SpecifyKind(t, DateTimeKind.Utc), + }; + /// /// The T0 Q9 statement for request times: $1 = tag, $(k+2) = time k. /// Columns: req, dir (0 = previous-or-equal, 1 = next), then ts, kind, num_value, int_value, str_value, diff --git a/src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/HistoryPaging.cs b/src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/HistoryPaging.cs index d1ffae37..6fa8c12b 100644 --- a/src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/HistoryPaging.cs +++ b/src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/HistoryPaging.cs @@ -212,25 +212,34 @@ internal static class HistoryPaging /// after (the reverse window excludes end; a resume at exactly end /// would turn into a start == end point read and return the excluded sample). Forward is unchanged. /// + /// + /// The backend's storage timestamp resolution (IHistorianDataSource.TimestampResolutionTicks, >= 1). A + /// drained cluster steps off by this many ticks rather than by one: a backend that + /// truncates its inputs to its resolution (QuestDB, 10 ticks) would otherwise truncate T + 1 tick back + /// to T and return the drained sample again, forever. Lossless for the same reason one tick is at tick + /// precision: no stored timestamp lies strictly between T and T ± resolution. The default + /// 1 keeps the tick-precision behaviour exactly. + /// public static void SliceTieCluster( int clusterCount, int skip, uint cap, DateTime boundaryT, DateTime endUtc, out int sliceStart, out int sliceCount, out DateTime? nextStartUtc, out int nextSkip, - bool reverse = false) + bool reverse = false, long resolutionTicks = 1) { + ArgumentOutOfRangeException.ThrowIfLessThan(resolutionTicks, 1L); sliceStart = Math.Min(skip, clusterCount); sliceCount = Math.Min((int)cap, clusterCount - sliceStart); var emitted = sliceStart + sliceCount; if (emitted < clusterCount) { nextStartUtc = boundaryT; nextSkip = emitted; } else if (reverse) { - // DateTime.MinValue cannot step down; nothing can lie below it anyway. - var more = boundaryT > DateTime.MinValue && boundaryT.AddTicks(-1) > endUtc; - nextStartUtc = more ? boundaryT.AddTicks(-1) : null; + // Nothing can lie below DateTime.MinValue, so a boundary within one resolution step of it cannot step down. + var more = boundaryT.Ticks >= resolutionTicks && boundaryT.AddTicks(-resolutionTicks) > endUtc; + nextStartUtc = more ? boundaryT.AddTicks(-resolutionTicks) : null; nextSkip = 0; } else { - var next = boundaryT.AddTicks(1); + var next = boundaryT.AddTicks(resolutionTicks); if (next <= endUtc) { nextStartUtc = next; nextSkip = 0; } else { nextStartUtc = null; nextSkip = 0; } } diff --git a/src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OtOpcUaNodeManager.cs b/src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OtOpcUaNodeManager.cs index 57f249c2..859b69b2 100644 --- a/src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OtOpcUaNodeManager.cs +++ b/src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OtOpcUaNodeManager.cs @@ -5931,7 +5931,7 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2 HistoryPaging.SliceTieCluster( cluster.Count, boundarySkip, numValuesPerNode, startUtc, endUtc, out var sliceStart, out var sliceCount, out var nextStartUtc, out var nextSkip, - reverse: reverse); + reverse: reverse, resolutionTicks: Math.Max(1L, HistorianDataSource.TimestampResolutionTicks)); var slice = new List(sliceCount); for (var i = sliceStart; i < sliceStart + sliceCount; i++) slice.Add(cluster[i]); diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/AtTimeRegressionTests.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/AtTimeRegressionTests.cs new file mode 100644 index 00000000..6f5a40fa --- /dev/null +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/AtTimeRegressionTests.cs @@ -0,0 +1,85 @@ +using Shouldly; +using Xunit; +using ZB.MOM.WW.OtOpcUa.Core.Abstractions; +using ZB.MOM.WW.OtOpcUa.Driver.Historian.Conformance; +using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common; + +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests; + +/// +/// QuestDB AtTime regressions found in the T5 review (M1, M3), and the data source's reported timestamp +/// resolution (the HistoryRead paging step, C1). +/// +[Collection(QuestDbCollection.Name)] +public sealed class AtTimeRegressionTests(QuestDbFixture fx) : IAsyncLifetime +{ + private static readonly CancellationToken Ct = CancellationToken.None; + private const uint GoodInterpolated = 0x00000402u; + + private QuestDbConformanceBackend _b = null!; + + private static DateTime At(double s) => ConformanceData.At(s); + + public async ValueTask InitializeAsync() => _b = await QuestDbConformanceBackend.CreateConcreteAsync(fx); + + public async ValueTask DisposeAsync() + { + if (_b is not null) await _b.DisposeAsync(); + } + + /// + /// M3: a request time with a sub-microsecond remainder is answered stamped with THAT time (as SQL Server does), + /// not with the rounded time the seek used. The value is still the one at the rounded time. + /// + [Fact] + public async Task AtTime_result_is_stamped_with_the_original_sub_microsecond_request_time() + { + await ConformanceData.SeedAAsync(_b); // +0=10, +10=20, ... + var requested = _b.Round(At(5)).AddTicks(3); // 0.3 µs past a microsecond (T0 itself is not µs-aligned) + var unspecified = DateTime.SpecifyKind(_b.Round(At(7)).AddTicks(7), DateTimeKind.Unspecified); + + var s = (await _b.CreateDataSource().ReadAtTimeAsync("A", [requested, unspecified], Ct)).Samples; + + s[0].SourceTimestampUtc.ShouldBe(requested); + s[0].SourceTimestampUtc!.Value.Kind.ShouldBe(DateTimeKind.Utc); + s[0].Value.ShouldBe(15d); // evaluated at the rounded time Round(At(5)) + s[0].StatusCode.ShouldBe(GoodInterpolated); + s[1].SourceTimestampUtc.ShouldBe(DateTime.SpecifyKind(unspecified, DateTimeKind.Utc)); + s[1].SourceTimestampUtc!.Value.Ticks.ShouldBe(unspecified.Ticks); + } + + /// M3 on the unknown-tag path: every BadNoData entry carries its own requested time. + [Fact] + public async Task AtTime_unknown_tag_entries_are_stamped_with_the_original_request_time() + { + var requested = _b.Round(At(5)).AddTicks(3); + var s = (await _b.CreateDataSource().ReadAtTimeAsync("nope", [requested], Ct)).Samples; + s.Single().StatusCode.ShouldBe(HistorianStatusCodes.BadNoData); + s.Single().SourceTimestampUtc.ShouldBe(requested); + } + + /// + /// M1: the NEXT seek skips a tombstone and finds the next live row. With +10 deleted, At(5) lies between the + /// live +0 (10) and the live +20 (30), so it interpolates to 15, Good. Seeing the tombstone as "next" would + /// instead hold 10 as Uncertain (an Empty BadNoData neighbour is not usable). + /// + [Fact] + public async Task AtTime_next_seek_skips_a_tombstone_and_finds_the_next_live_row() + { + await _b.SeedAsync("M", + (At(0), HistorianValue.FromDouble(10), ConformanceData.Good), + (At(10), HistorianValue.FromDouble(20), ConformanceData.Good), + (At(20), HistorianValue.FromDouble(30), ConformanceData.Good)); + await _b.InsertRowsAsync("M", [(At(10), HistorianValue.Empty, HistorianStatusCodes.BadNoData, true)]); + + var s = (await _b.CreateDataSource().ReadAtTimeAsync("M", [At(5)], Ct)).Samples.Single(); + + s.Value.ShouldBe(15d); // 10 + (30 - 10) * 5/20 + s.StatusCode.ShouldBe(GoodInterpolated); + } + + /// C1: the data source reports QuestDB's microsecond storage resolution to the paging code. + [Fact] + public void Data_source_reports_a_ten_tick_timestamp_resolution() => + ((IHistorianDataSource)_b.CreateDataSource()).TimestampResolutionTicks.ShouldBe(10); +} diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/QuestDbConformance.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/QuestDbConformance.cs new file mode 100644 index 00000000..63477891 --- /dev/null +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/QuestDbConformance.cs @@ -0,0 +1,55 @@ +using ZB.MOM.WW.OtOpcUa.Driver.Historian.Conformance; +using Xunit; + +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests; + +// One concrete subclass per conformance class: the suite's tests run here against QuestDB. + +[Collection(QuestDbCollection.Name)] +public sealed class QuestDbRawReadConformance(QuestDbFixture fx) : RawReadConformance +{ + /// + protected override Task CreateBackendAsync() => QuestDbConformanceBackend.CreateAsync(fx); +} + +[Collection(QuestDbCollection.Name)] +public sealed class QuestDbAtTimeConformance(QuestDbFixture fx) : AtTimeConformance +{ + /// + protected override Task CreateBackendAsync() => QuestDbConformanceBackend.CreateAsync(fx); +} + +[Collection(QuestDbCollection.Name)] +public sealed class QuestDbProcessedConformance(QuestDbFixture fx) : ProcessedConformance +{ + /// + protected override Task CreateBackendAsync() => QuestDbConformanceBackend.CreateAsync(fx); +} + +[Collection(QuestDbCollection.Name)] +public sealed class QuestDbQualityContractConformance(QuestDbFixture fx) : QualityContractConformance +{ + /// + protected override Task CreateBackendAsync() => QuestDbConformanceBackend.CreateAsync(fx); +} + +[Collection(QuestDbCollection.Name)] +public sealed class QuestDbLiveWriteConformance(QuestDbFixture fx) : LiveWriteConformance +{ + /// + protected override Task CreateBackendAsync() => QuestDbConformanceBackend.CreateAsync(fx); +} + +[Collection(QuestDbCollection.Name)] +public sealed class QuestDbUpdateConformance(QuestDbFixture fx) : UpdateConformance +{ + /// + protected override Task CreateBackendAsync() => QuestDbConformanceBackend.CreateAsync(fx); +} + +[Collection(QuestDbCollection.Name)] +public sealed class QuestDbTagNameConformance(QuestDbFixture fx) : TagNameConformance +{ + /// + protected override Task CreateBackendAsync() => QuestDbConformanceBackend.CreateAsync(fx); +} diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/QuestDbConformanceBackend.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/QuestDbConformanceBackend.cs index 5dfc6f4c..bf863363 100644 --- a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/QuestDbConformanceBackend.cs +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/QuestDbConformanceBackend.cs @@ -76,14 +76,13 @@ internal sealed class QuestDbConformanceBackend : IConformanceBackend new QuestDbHistorianDataSource(Conn, Boot, NewCatalog(), timeProvider: clock); /// - /// TODO(T7): build the T4 QuestDbHistorianValueWriter here once T4 is merged. + /// The T4 live-value writer over a fresh catalog (fresh positive cache) and this instance's WAL wait. public IHistorianValueWriter CreateValueWriter(Func? utcNow = null) => - throw new NotSupportedException("TODO(T7): the QuestDB value writer (T4) is not merged into this branch yet."); + new QuestDbHistorianValueWriter(Conn, NewCatalog(), Wal, utcNow: utcNow); /// - /// TODO(T7): build the T6 QuestDbHistorianDataUpdater here once T6 is merged. - public IHistorianDataUpdater CreateUpdater() => - throw new NotSupportedException("TODO(T7): the QuestDB history updater (T6) is not merged into this branch yet."); + /// The T6 soft-delete updater over a fresh catalog and this instance's WAL wait. + public IHistorianDataUpdater CreateUpdater() => new QuestDbHistorianDataUpdater(Conn, NewCatalog(), Wal); /// /// diff --git a/tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/HistoryPagingTests.cs b/tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/HistoryPagingTests.cs index b97d859a..aeab3c5e 100644 --- a/tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/HistoryPagingTests.cs +++ b/tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/HistoryPagingTests.cs @@ -345,6 +345,51 @@ public sealed class HistoryPagingTests store.TryTake(null, cp3).ShouldNotBeNull(); // retained } + [Fact] + public void SliceTieCluster_forward_drain_steps_by_the_backend_resolution() + { + // A microsecond backend (QuestDB, 10 ticks) truncates T + 1 tick back to T, so the drained cluster must be + // stepped off by the resolution, not by one tick, or the same sample comes back forever. + var t = new DateTime(2026, 1, 1, 0, 0, 5, DateTimeKind.Utc); + + HistoryPaging.SliceTieCluster( + clusterCount: 1, skip: 1, cap: 1, boundaryT: t, endUtc: t.AddHours(1), + out _, out var sliceCount, out var nextStartUtc, out var nextSkip, resolutionTicks: 10); + + sliceCount.ShouldBe(0); + nextStartUtc.ShouldBe(t.AddTicks(10)); + nextSkip.ShouldBe(0); + } + + [Fact] + public void SliceTieCluster_reverse_drain_steps_down_by_the_backend_resolution() + { + var t = new DateTime(2026, 1, 1, 0, 0, 5, DateTimeKind.Utc); + + HistoryPaging.SliceTieCluster( + clusterCount: 1, skip: 1, cap: 1, boundaryT: t, endUtc: t.AddHours(-1), + out _, out _, out var nextStartUtc, out _, reverse: true, resolutionTicks: 10); + + nextStartUtc.ShouldBe(t.AddTicks(-10)); + } + + [Fact] + public void SliceTieCluster_forward_drain_terminates_when_one_resolution_step_passes_the_end() + { + var t = new DateTime(2026, 1, 1, 0, 0, 5, DateTimeKind.Utc); + + HistoryPaging.SliceTieCluster( + clusterCount: 1, skip: 1, cap: 1, boundaryT: t, endUtc: t.AddTicks(9), + out _, out _, out var nextStartUtc, out _, resolutionTicks: 10); + + nextStartUtc.ShouldBeNull(); + } + + [Fact] + public void SliceTieCluster_rejects_a_resolution_below_one() => + Should.Throw(() => HistoryPaging.SliceTieCluster( + 1, 1, 1, DateTime.UtcNow, DateTime.UtcNow.AddHours(1), out _, out _, out _, out _, resolutionTicks: 0)); + private static HistoryContinuationState RawState() => new( "WW.Tag", DateTime.UtcNow, DateTime.UtcNow.AddHours(1), BoundarySkipCount: 1, NumValuesPerNode: 50); diff --git a/tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/NodeManagerHistoryReadPagingTests.cs b/tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/NodeManagerHistoryReadPagingTests.cs index b9388b55..80785796 100644 --- a/tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/NodeManagerHistoryReadPagingTests.cs +++ b/tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/NodeManagerHistoryReadPagingTests.cs @@ -357,6 +357,56 @@ public sealed class NodeManagerHistoryReadPagingTests : IDisposable await host.DisposeAsync(); } + /// + /// Forward paging at NumValuesPerNode = 1 over a MICROSECOND backend (QuestDB: it truncates every inbound + /// time to 10 ticks and reports TimestampResolutionTicks = 10) returns every sample exactly once and + /// terminates. Before the fix the drained one-sample "cluster" resumed at T + 1 tick, which the backend + /// truncated back to T: the same sample came back with no boundary skip, and the read looped forever. + /// + [Fact] + public async Task Raw_forward_read_with_one_value_per_page_on_a_microsecond_backend_returns_every_sample_once() + { + var (host, server) = await BootAsync(); + var nm = server.NodeManager!; + nm.HistoryContinuationStore = new InMemoryHistoryContinuationStore(); + // Microsecond-spaced samples, so a one-tick step would land strictly between two of them. + var series = Enumerable.Range(0, 6) + .Select(i => { var t = Epoch.AddTicks(10L * i); return new DataValueSnapshot((double)i, StatusCodes.Good, t, t); }) + .ToArray(); + nm.HistorianDataSource = new MicrosecondHistorianDataSource(series); + + nm.EnsureVariable("eq-1/us", parentFolderNodeId: null, displayName: "Us", dataType: "Double", + writable: false, historianTagname: "WW.Us", realm: AddressSpaceRealm.Uns); + var nodeId = nm.TryGetVariable("eq-1/us")!.NodeId; + + var collected = await DrainAsync(nm, nodeId, start: Epoch, end: Epoch.AddHours(1), max: 1); + + collected.ShouldBe([0d, 1d, 2d, 3d, 4d, 5d]); + await host.DisposeAsync(); + } + + /// The reverse mirror on the same microsecond backend: newest first, each sample once. + [Fact] + public async Task Raw_reverse_read_with_one_value_per_page_on_a_microsecond_backend_returns_every_sample_once() + { + var (host, server) = await BootAsync(); + var nm = server.NodeManager!; + nm.HistoryContinuationStore = new InMemoryHistoryContinuationStore(); + var series = Enumerable.Range(0, 6) + .Select(i => { var t = Epoch.AddTicks(10L * (i + 1)); return new DataValueSnapshot((double)i, StatusCodes.Good, t, t); }) + .ToArray(); + nm.HistorianDataSource = new MicrosecondHistorianDataSource(series); + + nm.EnsureVariable("eq-1/usrev", parentFolderNodeId: null, displayName: "UsRev", dataType: "Double", + writable: false, historianTagname: "WW.UsRev", realm: AddressSpaceRealm.Uns); + var nodeId = nm.TryGetVariable("eq-1/usrev")!.NodeId; + + var collected = await DrainAsync(nm, nodeId, start: Epoch.AddHours(1), end: Epoch, max: 1); + + collected.ShouldBe([5d, 4d, 3d, 2d, 1d, 0d]); + await host.DisposeAsync(); + } + /// Pages a Raw read to completion and returns the union of the pages in arrival order. private static Task> DrainAsync(OtOpcUaNodeManager nm, NodeId nodeId, DateTime start, DateTime end, uint max) { @@ -520,6 +570,43 @@ public sealed class NodeManagerHistoryReadPagingTests : IDisposable public void Dispose() { } } + /// + /// A microsecond-resolution backend, shaped like QuestDB: every inbound window bound is TRUNCATED to 10 ticks + /// (as QuestDbConnection.Round does) before the series is filtered, and the source reports + /// = 10. + /// + private sealed class MicrosecondHistorianDataSource(IReadOnlyList series) : IHistorianDataSource + { + private readonly SeriesHistorianDataSource _inner = new(series); + + public long TimestampResolutionTicks => 10; + + private static DateTime Truncate(DateTime t) => new(t.Ticks - t.Ticks % 10, DateTimeKind.Utc); + + public Task ReadRawAsync( + string fullReference, DateTime startUtc, DateTime endUtc, uint maxValuesPerNode, + CancellationToken cancellationToken) => + _inner.ReadRawAsync(fullReference, Truncate(startUtc), Truncate(endUtc), maxValuesPerNode, cancellationToken); + + public Task ReadProcessedAsync( + string fullReference, DateTime startUtc, DateTime endUtc, TimeSpan interval, + HistoryAggregateType aggregate, CancellationToken cancellationToken) => + _inner.ReadProcessedAsync(fullReference, startUtc, endUtc, interval, aggregate, cancellationToken); + + public Task ReadAtTimeAsync( + string fullReference, IReadOnlyList timestampsUtc, CancellationToken cancellationToken) => + _inner.ReadAtTimeAsync(fullReference, timestampsUtc, cancellationToken); + + public Task ReadEventsAsync( + string? sourceName, DateTime startUtc, DateTime endUtc, int maxEvents, + CancellationToken cancellationToken) => + _inner.ReadEventsAsync(sourceName, startUtc, endUtc, maxEvents, cancellationToken); + + public HistorianHealthSnapshot GetHealthSnapshot() => _inner.GetHealthSnapshot(); + + public void Dispose() { } + } + private async Task<(OpcUaApplicationHost Host, OtOpcUaSdkServer Server)> BootAsync() { var applicationUri = $"urn:OtOpcUa.HistoryPagingTest:{Guid.NewGuid():N}"; From 3244f9781a17e5100bb5204b48c6cd196bb3153b Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Sun, 11 Oct 2026 02:32:11 -0400 Subject: [PATCH 18/23] refactor(historian): one QuestDbTimeClusters helper for writer + updater existence reads; redelivery tombstone test Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe --- .../QuestDbHistorianDataUpdater.cs | 25 +------- .../QuestDbHistorianValueWriter.cs | 38 +----------- .../QuestDbTimeClusters.cs | 49 +++++++++++++++ .../TombstoneTests.Redelivery.cs | 38 ++++++++++++ .../TombstoneTests.cs | 2 +- .../UpdateTests.cs | 10 --- .../WriteTests.cs | 19 ------ .../QuestDbTimeClustersTests.cs | 62 +++++++++++++++++++ 8 files changed, 152 insertions(+), 91 deletions(-) create mode 100644 src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbTimeClusters.cs create mode 100644 tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/TombstoneTests.Redelivery.cs create mode 100644 tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests/QuestDbTimeClustersTests.cs diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianDataUpdater.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianDataUpdater.cs index 27e4f416..f4e00820 100644 --- a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianDataUpdater.cs +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianDataUpdater.cs @@ -31,9 +31,6 @@ internal sealed class QuestDbHistorianDataUpdater : IHistorianDataUpdater /// Rows per INSERT and per DeleteRaw page (T0 Q2: 9,000 parameters, far under the 32,767 limit). internal const int BatchSize = 1000; - /// Request times further apart than this are read with separate range queries (I-2). - internal static readonly TimeSpan ExistenceClusterGap = TimeSpan.FromMinutes(1); - private const long ResolutionTicks = 10; private const int LockStripes = 64; private readonly SemaphoreSlim[] _locks = Enumerable.Range(0, LockStripes).Select(_ => new SemaphoreSlim(1, 1)).ToArray(); @@ -60,26 +57,6 @@ internal sealed class QuestDbHistorianDataUpdater : IHistorianDataUpdater /// Rows returned by existence reads so far (test/diagnostic counter). internal long LiveRowsRead => Interlocked.Read(ref _liveRowsRead); - /// Splits times into clusters wherever consecutive sorted times are further apart than . - /// Request times (any order). - /// Largest gap kept inside one cluster. - /// Inclusive [Lo, Hi] ranges, ascending. - internal static List<(DateTime Lo, DateTime Hi)> Cluster(IReadOnlyList times, TimeSpan gap) - { - var result = new List<(DateTime, DateTime)>(); - if (times.Count == 0) return result; - var sorted = times.OrderBy(t => t).ToList(); - var lo = sorted[0]; - var prev = lo; - foreach (var t in sorted.Skip(1)) - { - if (t - prev > gap) { result.Add((lo, prev)); lo = t; } - prev = t; - } - result.Add((lo, prev)); - return result; - } - private async Task LockAsync(string tag, CancellationToken ct) { var sem = _locks[(int)((uint)StringComparer.Ordinal.GetHashCode(tag) % LockStripes)]; @@ -231,7 +208,7 @@ internal sealed class QuestDbHistorianDataUpdater : IHistorianDataUpdater { var live = new HashSet(); await using var conn = await _conn.OpenAsync(ct).ConfigureAwait(false); - foreach (var (lo, hi) in Cluster(times, ExistenceClusterGap)) + foreach (var (lo, hi) in QuestDbTimeClusters.Cluster(times, QuestDbTimeClusters.ExistenceGap)) { await using var cmd = _conn.CreateCommand(conn, $"SELECT ts FROM {_conn.SampleTable} WHERE tag = $1 AND ts >= $2 AND ts <= $3 AND NOT deleted"); diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianValueWriter.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianValueWriter.cs index 791ef010..a7fba836 100644 --- a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianValueWriter.cs +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianValueWriter.cs @@ -98,49 +98,13 @@ internal sealed class QuestDbHistorianValueWriter : IHistorianValueWriter } } - /// A gap between consecutive batch times larger than this starts a new existence-query range. - internal static readonly TimeSpan ExistenceClusterGap = TimeSpan.FromMinutes(1); - - /// - /// Splits ascending into inclusive [Lo, Hi] ranges: a new range starts where the gap to the - /// previous time exceeds or the range already holds times. One outlier - /// time therefore costs its own tiny query instead of widening one range over every stored row in between. - /// - /// Times in ascending order. - /// Largest gap kept inside one range. - /// Most times in one range (>= 1). - /// The ranges, ascending. - internal static IReadOnlyList<(DateTime Lo, DateTime Hi)> Cluster(IReadOnlyList sortedTimes, TimeSpan gap, int maxPerCluster) - { - ArgumentOutOfRangeException.ThrowIfLessThan(maxPerCluster, 1); - var result = new List<(DateTime, DateTime)>(); - if (sortedTimes.Count == 0) return result; - var lo = sortedTimes[0]; - var prev = lo; - var n = 1; - for (var i = 1; i < sortedTimes.Count; i++) - { - var t = sortedTimes[i]; - if (t - prev > gap || n >= maxPerCluster) - { - result.Add((lo, prev)); - lo = t; - n = 0; - } - prev = t; - n++; - } - result.Add((lo, prev)); - return result; - } - // T0 Q11 range form, one query per cluster: tombstones are NOT filtered out (a deleted row must not be resurrected). private async Task> ExistingTicksAsync(string tag, List rows, CancellationToken ct) { var times = rows.Select(r => QuestDbConnection.Round(r.TimeUtc)).OrderBy(t => t).ToList(); var set = new HashSet(); await using var conn = await _conn.OpenAsync(ct).ConfigureAwait(false); - foreach (var (lo, hi) in Cluster(times, ExistenceClusterGap, BatchRows)) + foreach (var (lo, hi) in QuestDbTimeClusters.Cluster(times, QuestDbTimeClusters.ExistenceGap, BatchRows)) { await using var cmd = _conn.CreateCommand(conn, $"SELECT ts FROM {_conn.SampleTable} WHERE tag = $1 AND ts >= $2 AND ts <= $3"); QuestDbConnection.BindText(cmd, tag); diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbTimeClusters.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbTimeClusters.cs new file mode 100644 index 00000000..442e7b34 --- /dev/null +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbTimeClusters.cs @@ -0,0 +1,49 @@ +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb; + +/// +/// Groups the times of one write or HistoryUpdate request into inclusive [Lo, Hi] ranges so that the +/// "which of these times already exist" check reads one range query per cluster instead of one query spanning +/// [min, max] of the whole request. One far-away outlier time then costs its own tiny query instead of +/// widening a single range over every stored row in between. Shared by the live-value writer and the updater. +/// +internal static class QuestDbTimeClusters +{ + /// Request times further apart than this start a new existence-query range. + internal static readonly TimeSpan ExistenceGap = TimeSpan.FromMinutes(1); + + /// + /// Splits (any order; sorted here) into inclusive ranges, ascending: a new range starts + /// where the gap to the previous time exceeds (exactly stays + /// together), or where the current range already holds times. + /// + /// Request times, in any order. + /// Largest gap kept inside one range. + /// Most times in one range (>= 1); unbounded by default. + /// The ranges, ascending; empty for no times. + internal static IReadOnlyList<(DateTime Lo, DateTime Hi)> Cluster( + IReadOnlyList times, TimeSpan gap, int maxPerCluster = int.MaxValue) + { + ArgumentNullException.ThrowIfNull(times); + ArgumentOutOfRangeException.ThrowIfLessThan(maxPerCluster, 1); + var result = new List<(DateTime, DateTime)>(); + if (times.Count == 0) return result; + var sorted = times.Order().ToList(); + var lo = sorted[0]; + var prev = lo; + var n = 1; + for (var i = 1; i < sorted.Count; i++) + { + var t = sorted[i]; + if (t - prev > gap || n >= maxPerCluster) + { + result.Add((lo, prev)); + lo = t; + n = 0; + } + prev = t; + n++; + } + result.Add((lo, prev)); + return result; + } +} diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/TombstoneTests.Redelivery.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/TombstoneTests.Redelivery.cs new file mode 100644 index 00000000..c3b85295 --- /dev/null +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/TombstoneTests.Redelivery.cs @@ -0,0 +1,38 @@ +using Shouldly; +using Xunit; +using ZB.MOM.WW.OtOpcUa.Core.Abstractions; +using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common; + +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests; + +public sealed partial class TombstoneTests +{ + /// + /// The live writer's outbox redelivers batches (at-least-once). A batch replayed after an operator's + /// DeleteAtTime must not bring the deleted sample back: the writer's existence check counts a tombstone as + /// present (it deliberately does not filter NOT deleted), so the redelivered row is skipped. + /// + [Fact] + public async Task Tombstone_is_not_resurrected_by_a_writer_redelivery() + { + fx.SkipIfUnavailable(); + var rig = UpdateTests.MakeRig(fx); + var writer = new QuestDbHistorianValueWriter(rig.Conn, rig.Tags, + new QuestDbWalVisibility(rig.Conn, TimeSpan.FromSeconds(5))); + HistorizationValue[] batch = + [ + new(T0, HistorianValue.FromDouble(1), HistorianStatusCodes.Good), + new(T0.AddSeconds(1), HistorianValue.FromDouble(2), HistorianStatusCodes.Good), + ]; + + (await writer.WriteLiveValuesAsync("r", batch, default)).ShouldBeTrue(); + (await rig.Updater.DeleteAtTimeAsync("r", [T0], default)).ShouldBe([HistorianStatusCodes.Good]); + (await writer.WriteLiveValuesAsync("r", batch, default)).ShouldBeTrue(); // the redelivery + + (await UpdateTests.RowsAsync(rig, "r", liveOnly: true)).Select(r => r.Num).ShouldBe([2d]); + var all = await UpdateTests.RowsAsync(rig, "r"); + all.Count.ShouldBe(2); + all[0].Deleted.ShouldBeTrue(); + all[0].Num.ShouldBeNull(); + } +} diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/TombstoneTests.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/TombstoneTests.cs index 2f3f53e3..841b73b7 100644 --- a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/TombstoneTests.cs +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/TombstoneTests.cs @@ -7,7 +7,7 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests; /// Soft-delete behaviour. The writer-redelivery case lives in T7 (TombstoneTests.Redelivery.cs). [Collection(QuestDbCollection.Name)] -public sealed class TombstoneTests(QuestDbFixture fx) +public sealed partial class TombstoneTests(QuestDbFixture fx) { private static readonly DateTime T0 = new(2026, 3, 1, 12, 0, 0, DateTimeKind.Utc); diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/UpdateTests.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/UpdateTests.cs index 0f721065..4d4e52a5 100644 --- a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/UpdateTests.cs +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/UpdateTests.cs @@ -249,16 +249,6 @@ public sealed class UpdateTests(QuestDbFixture fx) } } - [Fact] - public void Cluster_splits_where_the_gap_exceeds_the_limit() - { - var gap = QuestDbHistorianDataUpdater.ExistenceClusterGap; - var c = QuestDbHistorianDataUpdater.Cluster( - [T0.AddHours(2), T0, T0.AddSeconds(30), T0.AddHours(1)], gap); - c.ShouldBe([(T0, T0.AddSeconds(30)), (T0.AddHours(1), T0.AddHours(1)), (T0.AddHours(2), T0.AddHours(2))]); - QuestDbHistorianDataUpdater.Cluster([], gap).ShouldBeEmpty(); - } - [Fact] public async Task Existence_reads_are_bounded_when_request_times_are_far_apart() { diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/WriteTests.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/WriteTests.cs index 39e75e1b..a090346f 100644 --- a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/WriteTests.cs +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/WriteTests.cs @@ -274,23 +274,4 @@ public sealed class WriteTests(QuestDbFixture fx) (await w.WriteLiveValuesAsync("tag.far", vals, CancellationToken.None)).ShouldBeTrue(); (await RowsAsync(c, "tag.far")).Count.ShouldBe(2); } - - [Fact] - public void Cluster_splits_on_a_gap_over_the_limit_and_at_the_per_cluster_cap() - { - var g = TimeSpan.FromMinutes(1); - QuestDbHistorianValueWriter.Cluster([], g, 10).ShouldBeEmpty(); - QuestDbHistorianValueWriter.Cluster([T0], g, 10).ShouldBe([(T0, T0)]); - // exactly the gap stays together; one tick over splits - QuestDbHistorianValueWriter.Cluster([T0, T0 + g], g, 10).ShouldBe([(T0, T0 + g)]); - QuestDbHistorianValueWriter.Cluster([T0, T0 + g + TimeSpan.FromTicks(10)], g, 10) - .ShouldBe([(T0, T0), (T0 + g + TimeSpan.FromTicks(10), T0 + g + TimeSpan.FromTicks(10))]); - // outlier gets its own range - var far = T0.AddYears(1); - QuestDbHistorianValueWriter.Cluster([T0, T0.AddSeconds(1), far], g, 10).ShouldBe([(T0, T0.AddSeconds(1)), (far, far)]); - // cap: 5 close times, max 2 per cluster - var five = Enumerable.Range(0, 5).Select(i => T0.AddSeconds(i)).ToList(); - QuestDbHistorianValueWriter.Cluster(five, g, 2) - .ShouldBe([(five[0], five[1]), (five[2], five[3]), (five[4], five[4])]); - } } diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests/QuestDbTimeClustersTests.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests/QuestDbTimeClustersTests.cs new file mode 100644 index 00000000..f68f9521 --- /dev/null +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests/QuestDbTimeClustersTests.cs @@ -0,0 +1,62 @@ +using Shouldly; +using Xunit; + +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests; + +/// +/// The shared existence-query clustering (moved here, names unchanged, from the writer's WriteTests and the +/// updater's UpdateTests when the two private copies were consolidated into ). +/// +public sealed class QuestDbTimeClustersTests +{ + private static readonly DateTime T0 = new(2026, 3, 1, 12, 0, 0, DateTimeKind.Utc); + + [Fact] + public void Cluster_splits_on_a_gap_over_the_limit_and_at_the_per_cluster_cap() + { + var g = TimeSpan.FromMinutes(1); + QuestDbTimeClusters.Cluster([], g, 10).ShouldBeEmpty(); + QuestDbTimeClusters.Cluster([T0], g, 10).ShouldBe([(T0, T0)]); + // exactly the gap stays together; one tick over splits + QuestDbTimeClusters.Cluster([T0, T0 + g], g, 10).ShouldBe([(T0, T0 + g)]); + QuestDbTimeClusters.Cluster([T0, T0 + g + TimeSpan.FromTicks(10)], g, 10) + .ShouldBe([(T0, T0), (T0 + g + TimeSpan.FromTicks(10), T0 + g + TimeSpan.FromTicks(10))]); + // outlier gets its own range + var far = T0.AddYears(1); + QuestDbTimeClusters.Cluster([T0, T0.AddSeconds(1), far], g, 10).ShouldBe([(T0, T0.AddSeconds(1)), (far, far)]); + // cap: 5 close times, max 2 per cluster + var five = Enumerable.Range(0, 5).Select(i => T0.AddSeconds(i)).ToList(); + QuestDbTimeClusters.Cluster(five, g, 2) + .ShouldBe([(five[0], five[1]), (five[2], five[3]), (five[4], five[4])]); + } + + [Fact] + public void Cluster_splits_where_the_gap_exceeds_the_limit() + { + var gap = QuestDbTimeClusters.ExistenceGap; + var c = QuestDbTimeClusters.Cluster( + [T0.AddHours(2), T0, T0.AddSeconds(30), T0.AddHours(1)], gap); + c.ShouldBe([(T0, T0.AddSeconds(30)), (T0.AddHours(1), T0.AddHours(1)), (T0.AddHours(2), T0.AddHours(2))]); + QuestDbTimeClusters.Cluster([], gap).ShouldBeEmpty(); + } + + [Fact] + public void The_existence_gap_is_one_minute_and_the_default_cap_is_unbounded() + { + QuestDbTimeClusters.ExistenceGap.ShouldBe(TimeSpan.FromMinutes(1)); + var many = Enumerable.Range(0, 5000).Select(i => T0.AddMilliseconds(i)).ToList(); + QuestDbTimeClusters.Cluster(many, QuestDbTimeClusters.ExistenceGap).ShouldBe([(many[0], many[^1])]); + } + + [Fact] + public void Unsorted_input_is_sorted_before_capping() + { + var times = new[] { T0.AddSeconds(3), T0, T0.AddSeconds(2), T0.AddSeconds(1) }; + QuestDbTimeClusters.Cluster(times, TimeSpan.FromMinutes(1), 2) + .ShouldBe([(T0, T0.AddSeconds(1)), (T0.AddSeconds(2), T0.AddSeconds(3))]); + } + + [Fact] + public void A_cap_below_one_is_rejected() => + Should.Throw(() => QuestDbTimeClusters.Cluster([T0], TimeSpan.FromMinutes(1), 0)); +} From 0e2bffc74a887a81e839a6c2798880c9ecc71109 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Sun, 11 Oct 2026 02:53:40 -0400 Subject: [PATCH 19/23] feat(historian): QuestDb provider + Host wiring; conformance suite green on both backends AtTime conformance stamps are the requested time (M3; identical on SQL Server). The QuestDB test fixture prunes idle pooled connections after 1 s: undisposed per-test data sources exhausted QuestDB's 64-connection PG-wire limit in a full run. Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe --- .../Historian.Conformance/AtTimeConformance.cs | 10 +++++----- .../QuestDbFixture.cs | 12 +++++++++++- .../ReadTests.cs | 2 +- 3 files changed, 17 insertions(+), 7 deletions(-) diff --git a/tests/Drivers/Historian.Conformance/AtTimeConformance.cs b/tests/Drivers/Historian.Conformance/AtTimeConformance.cs index 8a2acea7..2f8a9341 100644 --- a/tests/Drivers/Historian.Conformance/AtTimeConformance.cs +++ b/tests/Drivers/Historian.Conformance/AtTimeConformance.cs @@ -35,7 +35,7 @@ public abstract class AtTimeConformance : ConformanceBase var s = (await B.CreateDataSource().ReadAtTimeAsync("A", [At(10), At(20)], Ct)).Samples; s[0].Value.ShouldBe(20d); s[0].StatusCode.ShouldBe(0u); - s[0].SourceTimestampUtc.ShouldBe(B.Round(At(10))); + s[0].SourceTimestampUtc.ShouldBe(At(10)); // the requested time itself, never rounded (M3) s[0].ServerTimestampUtc.ShouldBe(B.Round(At(11))); // the stored ServerTimeUtc s[1].Value.ShouldBe(30d); // a stored Bad sample comes back as stored s[1].StatusCode.ShouldBe(0x80000000u); @@ -50,7 +50,7 @@ public abstract class AtTimeConformance : ConformanceBase s[1].Value.ShouldBe(12.5d); // 10 + 10 * 2.5/10 s[2].Value.ShouldBe(42.5d); // 40 + 10 * 2.5/10 s.ShouldAllBe(x => x.StatusCode == GoodInterpolated); - s[0].SourceTimestampUtc.ShouldBe(B.Round(At(5))); // stamped at the requested (sub-ms) time + s[0].SourceTimestampUtc.ShouldBe(At(5)); // stamped at the requested (sub-µs) time itself, never rounded (M3) } [Fact] @@ -61,7 +61,7 @@ public abstract class AtTimeConformance : ConformanceBase var s = (await B.CreateDataSource().ReadAtTimeAsync("A", [before], Ct)).Samples.Single(); s.Value.ShouldBeNull(); s.StatusCode.ShouldBe(BadNoData); - s.SourceTimestampUtc.ShouldBe(B.Round(before)); + s.SourceTimestampUtc.ShouldBe(before); } [Fact] @@ -80,7 +80,7 @@ public abstract class AtTimeConformance : ConformanceBase var s = (await B.CreateDataSource().ReadAtTimeAsync("A", [At(25)], Ct)).Samples.Single(); s.Value.ShouldBeNull(); // NOT 35 (30→40) and NOT 30 (a Bad value) s.StatusCode.ShouldBe(0x80000000u); // the previous sample's Bad code — never upgraded to Uncertain - s.SourceTimestampUtc.ShouldBe(B.Round(At(25))); + s.SourceTimestampUtc.ShouldBe(At(25)); } [Fact] @@ -173,7 +173,7 @@ public abstract class AtTimeConformance : ConformanceBase var s = (await B.CreateDataSource().ReadAtTimeAsync("A", [At(5), At(-1), At(5), At(10)], Ct)).Samples; s.Count.ShouldBe(4); s.Select(x => x.Value).ShouldBe([15d, null, 15d, 20d]); - s.Select(x => x.SourceTimestampUtc).ShouldBe([B.Round(At(5)), B.Round(At(-1)), B.Round(At(5)), B.Round(At(10))]); + s.Select(x => x.SourceTimestampUtc).ShouldBe([At(5), At(-1), At(5), At(10)]); } [Fact] diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/QuestDbFixture.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/QuestDbFixture.cs index dcaba660..4d9d0877 100644 --- a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/QuestDbFixture.cs +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/QuestDbFixture.cs @@ -43,9 +43,19 @@ public sealed class QuestDbFixture : IAsyncLifetime /// Optional TTL. /// AutoCreateSchema. /// Valid options. + /// + /// Pooled connections idle for more than a second are closed (Connection Idle Lifetime / + /// Connection Pruning Interval). Many tests build their own (one Npgsql + /// data source each) and never dispose it; with Npgsql's default 300 s idle lifetime their pooled connections + /// piled up until QuestDB's PG-wire limit (64) was reached and later tests' opens timed out after 15 s. + /// public QuestDbHistorianOptions Options(string prefix, int? retentionDays = null, bool autoCreate = true) => new() { - ConnectionString = ConnectionString, + ConnectionString = new NpgsqlConnectionStringBuilder(ConnectionString) + { + ConnectionIdleLifetime = 1, + ConnectionPruningInterval = 1, + }.ConnectionString, TablePrefix = prefix, RetentionDays = retentionDays, AutoCreateSchema = autoCreate, diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/ReadTests.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/ReadTests.cs index d0f29d5a..6345613b 100644 --- a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/ReadTests.cs +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/ReadTests.cs @@ -184,7 +184,7 @@ public sealed class ReadTests(QuestDbFixture fx) : IAsyncLifetime times.Add(times[0]); var s = (await _b.CreateDataSource().ReadAtTimeAsync("A", times, Ct)).Samples; s.Count.ShouldBe(times.Count); - s.Select(x => x.SourceTimestampUtc).ShouldBe(times.Select(t => (DateTime?)_b.Round(t))); + s.Select(x => x.SourceTimestampUtc).ShouldBe(times.Select(t => (DateTime?)t)); // the requested times (M3) s[0].Value.ShouldBe(49d); // +39: 40 + 10 × 9/10 s[0].StatusCode.ShouldBe(GoodInterpolated); s[^1].Value.ShouldBe(49d); // the duplicate From f661c084d8c335e66c1c290fb1950037aa3b1cb5 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Sun, 11 Oct 2026 03:03:13 -0400 Subject: [PATCH 20/23] test(historian): QuestDB-backed paging regression (T7 review) Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe --- .../PagingRegressionTests.cs | 118 ++++++++++++++++++ 1 file changed, 118 insertions(+) create mode 100644 tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/PagingRegressionTests.cs diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/PagingRegressionTests.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/PagingRegressionTests.cs new file mode 100644 index 00000000..642c6b8f --- /dev/null +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/PagingRegressionTests.cs @@ -0,0 +1,118 @@ +using Shouldly; +using Xunit; +using ZB.MOM.WW.OtOpcUa.Core.Abstractions; +using ZB.MOM.WW.OtOpcUa.Driver.Historian.Conformance; + +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests; + +/// +/// C1 (T7 review I-1): the facts the server's HistoryRead-Raw paging relies on, pinned against the REAL QuestDB data +/// source (the node-manager walk in NodeManagerHistoryReadPagingTests runs on a fake that models them). +/// HistoryPaging is internal to OpcUaServer and not visible from this project, so the end-to-end walk below +/// re-states the node manager's cursor arithmetic (resume inclusive + boundary trim; stalled ⇒ point-read the +/// cluster and step off by ) over the real reads. +/// +[Collection(QuestDbCollection.Name)] +public sealed class PagingRegressionTests(QuestDbFixture fx) : IAsyncLifetime +{ + private static readonly CancellationToken Ct = CancellationToken.None; + private static readonly DateTime T0 = new(2026, 3, 1, 12, 0, 0, DateTimeKind.Utc); // µs-aligned + private QuestDbConformanceBackend _b = null!; + + /// Sample i at T0 + i µs (adjacent microseconds: a one-tick step lands strictly between two of them). + private static DateTime S(int i) => T0.AddTicks(10L * i); + + public async ValueTask InitializeAsync() + { + _b = await QuestDbConformanceBackend.CreateConcreteAsync(fx); + await _b.SeedAsync("P", Enumerable.Range(0, 6) + .Select(i => (S(i), HistorianValue.FromDouble(i), ConformanceData.Good)).ToArray()); + } + + public async ValueTask DisposeAsync() + { + if (_b is not null) await _b.DisposeAsync(); + } + + private async Task RawAsync(IHistorianDataSource ds, DateTime start, DateTime end, uint max) => + (await ds.ReadRawAsync("P", start, end, max, Ct)).Samples.Select(s => s.SourceTimestampUtc).ToArray(); + + [Fact] + public async Task Forward_raw_reads_truncate_one_tick_but_advance_by_one_resolution_step() + { + var ds = _b.CreateDataSource(); + var end = T0.AddHours(1); + var res = ds.TimestampResolutionTicks; + + (await RawAsync(ds, S(2).AddTicks(1), end, 1)).ShouldBe([S(2)]); // T + 1 tick ⇒ back to T (the C1 loop) + (await RawAsync(ds, S(2).AddTicks(res), end, 1)).ShouldBe([S(3)]); // T + resolution ⇒ the next sample + (await RawAsync(ds, S(2), S(2), 10)).ShouldBe([S(2)]); // point read at T ⇒ T + } + + [Fact] + public async Task Reverse_raw_reads_step_down_by_one_resolution_step() + { + var ds = _b.CreateDataSource(); + var end = T0.AddHours(-1); + var res = ds.TimestampResolutionTicks; + + (await RawAsync(ds, S(3).AddTicks(-1), end, 1)).ShouldBe([S(2)]); // T − 1 tick truncates DOWN ⇒ previous + (await RawAsync(ds, S(3).AddTicks(-res), end, 1)).ShouldBe([S(2)]); // T − resolution ⇒ previous + (await RawAsync(ds, S(3), end, 1)).ShouldBe([S(3)]); // reverse window (end, start] includes start + } + + [Theory] + [InlineData(false)] + [InlineData(true)] + public async Task One_value_per_page_walk_returns_every_sample_once_and_terminates(bool reverse) + { + var ds = _b.CreateDataSource(); + var (start, end) = reverse ? (T0.AddHours(1), T0.AddHours(-1)) : (T0.AddHours(-1), T0.AddHours(1)); + var collected = await WalkAsync(ds, start, end); + var expected = Enumerable.Range(0, 6).Select(i => (DateTime?)S(i)); + collected.ShouldBe(reverse ? expected.Reverse() : expected); + } + + /// The node manager's Raw paging loop at NumValuesPerNode = 1 (see ServeRawPagedAsync). + private static async Task> WalkAsync(IHistorianDataSource ds, DateTime start, DateTime end) + { + const uint cap = 1; + var res = ds.TimestampResolutionTicks; + var reverse = start > end; + var collected = new List(); + DateTime? cursor = start; + var skip = 0; + var resumed = false; + for (var pages = 0; cursor is { } from; pages++) + { + pages.ShouldBeLessThan(60, "paging must terminate, not loop"); + var raw = (await ds.ReadRawAsync("P", from, end, cap, Ct)).Samples; + var full = raw.Count >= cap; + var page = raw.ToList(); + if (resumed) + for (var dropped = 0; dropped < skip && page.Count > 0 && page[0].SourceTimestampUtc == from; dropped++) + page.RemoveAt(0); + + if (resumed && full && (page.Count == 0 || page[^1].SourceTimestampUtc == from)) + { + // Stalled on the boundary "cluster": point-read it, emit what is left, step off by the resolution. + var cluster = (await ds.ReadRawAsync("P", from, from, 1000, Ct)).Samples; + var sliceStart = Math.Min(skip, cluster.Count); + var slice = cluster.Skip(sliceStart).Take((int)cap).ToList(); + collected.AddRange(slice.Select(s => s.SourceTimestampUtc)); + if (sliceStart + slice.Count < cluster.Count) { skip = sliceStart + slice.Count; continue; } + var next = from.AddTicks(reverse ? -res : res); + cursor = reverse ? (next > end ? next : null) : (next <= end ? next : null); + skip = 0; + continue; + } + + collected.AddRange(page.Select(s => s.SourceTimestampUtc)); + if (!full || page.Count == 0) break; + cursor = page[^1].SourceTimestampUtc; + skip = page.Count(s => s.SourceTimestampUtc == cursor); + resumed = true; + } + return collected; + } +} From 94691ec50f29ac96903bfa3ee905ab82239b244c Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Sun, 11 Oct 2026 03:09:11 -0400 Subject: [PATCH 21/23] docs(historian): QuestDB backend documentation (as built) Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe --- CLAUDE.md | 5 +- docker-dev/README.md | 27 ++ docs/Configuration.md | 30 ++- docs/Historian.md | 234 ++++++++++++++++-- .../2026-10-10-questdb-historian-design.md | 103 +++++--- 5 files changed, 342 insertions(+), 57 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index cb21e377..baa899c2 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1325,7 +1325,7 @@ are no longer supported (pinned by a test). Condition teardown must forget the w ## Historian / HistoryRead -**Backend: pluggable — `ServerHistorian:Backend` = `Gateway` (default) or `SqlServer`** (see the subsection below). +**Backend: pluggable — `ServerHistorian:Backend` = `Gateway` (default), `SqlServer` or `QuestDb`** (see the subsection below). As of the gateway-integration cutover, the **Gateway** backend's historian read, alarm-write, and continuous-historization paths are all served by the **`ZB.MOM.WW.HistorianGateway`** sidecar, consumed as the Gitea-feed **`ZB.MOM.WW.HistorianGateway.Client`** gRPC package (`historian_gateway.v1`) behind a thin @@ -1340,8 +1340,9 @@ historian tagname, which is the tag's driver `FullName`. ### Pluggable backends, SQL Server and HistoryUpdate (2026-10) -- **Selection:** `ServerHistorian:Backend` (`Gateway` default | `SqlServer`; unknown fails host start). `HistorianBackendRegistry` → `IHistorianBackendProvider` builds all five seams; the gateway keys are only required for `Gateway`. `SqlServer` uses `ServerHistorian:SqlServer:{ConnectionString (required), Schema=hist, CommandTimeout=30s, AutoCreateSchema=true}`. +- **Selection:** `ServerHistorian:Backend` (`Gateway` default | `SqlServer` | `QuestDb`; unknown fails host start). `HistorianBackendRegistry` → `IHistorianBackendProvider` builds all five seams; the gateway keys are only required for `Gateway`. `SqlServer` uses `ServerHistorian:SqlServer:{ConnectionString (required), Schema=hist, CommandTimeout=30s, AutoCreateSchema=true}`. - **SQL Server** (`Driver.Historian.SqlServer`): `hist.Sample` PK `(TagId, SourceTimeUtc)` — writes are insert-if-absent, so outbox redelivery is idempotent. Stores every value kind with its **real status**; aggregates (13, simple per-sample definitions, not Part 13) are pushed down to T-SQL with bucket math in **100 ns ticks**. Docker-dev recipe, schema and aggregate semantics: `docs/Historian.md` §"SQL Server backend". Integration tests need `HISTORIAN_SQL_TEST_CONNECTION`. +- **QuestDB** (`Driver.Historian.QuestDb`, `ServerHistorian:QuestDb:*`, Npgsql over PG wire; docker-dev overlay `docker-compose.questdb.yml`): same kinds/status/aggregates/HistoryUpdate as SqlServer, soft delete by tombstone (INSERT only — never `UPDATE`/`DELETE`; every read filters `NOT deleted`). ⚠️ **Timestamps are µs** (`QuestDbConnection.Round`/`BindTimestamp`; `IHistorianDataSource.TimestampResolutionTicks` = 10 steps HistoryRead paging), **pre-1970 cannot be stored** (writer drops, HistoryUpdate `BadOutOfRange`), every write waits for WAL visibility, TTL is enforced on write commits only, set Npgsql `Maximum Pool Size` < QuestDB's 64. `docs/Historian.md` §"QuestDB backend". - **Real status fix:** the recorder used to stamp every sample Good (`192`); it now carries the driver's status. ⚠️ On the **Gateway** backend this changes nothing stored: the gateway discards quality (C-002), so AVEVA still stores Good. - **Comms loss is historized as an outage marker** (`Empty` + `BadNoCommunication`, one per tag per outage) on SqlServer; dropped (`DroppedUnsupportedKind`) on Gateway. Outbox records are **format v2** — drain or delete `ContinuousHistorization:OutboxPath` before downgrading a node. - **HistoryUpdate** (SqlServer only; Gateway ⇒ `BadHistoryOperationUnsupported`): Insert/Replace/Update/DeleteRaw/DeleteAtTime, gated in both ACL modes — flag off needs the `HistoryUpdate` role from a **Cluster-scoped** grant (+ key re-check), flag on uses `DataPlaneAuthorizer`'s HistoryUpdate operation; `WriteOperate` never implies it. The `HistoryWrite` AccessLevel bit is set only when an updater is wired at materialise time. `docs/Historian.md` §HistoryUpdate, `docs/security.md`. diff --git a/docker-dev/README.md b/docker-dev/README.md index f7706820..6dbdbcb6 100644 --- a/docker-dev/README.md +++ b/docker-dev/README.md @@ -160,6 +160,33 @@ and the script fails loudly saying so. A plain `docker compose up -d` also hides rig without the gate helping: `up -d` blocks on the `depends_on: service_healthy` conditions and returns ~30 s *after* Traefik starts, by which time the router table is long loaded. +## QuestDB historian overlay + +`docker-compose.questdb.yml` is an **optional overlay** (not part of the default rig) that adds one service, `questdb` +(`questdb/questdb:10.0.1`, container `otopcua-dev-questdb-1`, named volume `otopcua-questdb-data`), for the `ServerHistorian:Backend=QuestDb` +historian ([`docs/Historian.md`](../docs/Historian.md#questdb-backend)). Host ports: PG wire **`18812`** (→ 8812) and the web console / REST **`19000`** +(→ 9000), chosen so they do not collide with AdminUI `:9000`/`:9200`. + +```bash +# Bring up ONLY questdb (--no-deps: nothing else is touched): +docker compose -f docker-dev/docker-compose.yml -f docker-dev/docker-compose.questdb.yml up -d --no-deps questdb + +# Point the central nodes at it (the QuestDb connection string already defaults to the `questdb` service; +# override with OTOPCUA_HISTORIAN_QUESTDB). Keep the overlay -f flag on every compose command for these nodes: +OTOPCUA_HISTORIAN_ENABLED=true OTOPCUA_HISTORIAN_BACKEND=QuestDb OTOPCUA_CONTHIST_ENABLED=true \ + docker compose -f docker-dev/docker-compose.yml -f docker-dev/docker-compose.questdb.yml up -d central-1 central-2 + +# Inspect from the host (REST, no client needed): +curl -G localhost:19000/exec --data-urlencode "query=SELECT tag, count() n FROM hist_sample WHERE NOT deleted GROUP BY tag" + +# Integration tests (skipped when the variable is absent): +export HISTORIAN_QUESTDB_TEST_CONNECTION='Host=localhost;Port=18812;Username=admin;Password=quest;Database=qdb;ServerCompatibilityMode=NoTypeLoading' +dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests +``` + +Tear down with `docker compose -f docker-dev/docker-compose.yml -f docker-dev/docker-compose.questdb.yml rm -sf questdb` (add `docker volume rm otopcua-questdb-data` +to drop the data). The `admin`/`quest` credentials are QuestDB's stock defaults, not secrets. + ## Tear down ```bash diff --git a/docs/Configuration.md b/docs/Configuration.md index 54e3bb5b..597fa8cc 100644 --- a/docs/Configuration.md +++ b/docs/Configuration.md @@ -377,11 +377,13 @@ The Galaxy/MxAccess connection settings are **not an `appsettings` section.** Th > The `OTOPCUA_GALAXY_*` environment variables that v1's in-process `Galaxy.Host` consumed **no longer live in this repo** — they moved into the separately-installed mxaccessgw gateway's own config (see the v1 archive pointer in `docs/README.md` and the Galaxy overview at [`docs/drivers/Galaxy.md`](drivers/Galaxy.md)). The only Galaxy connection secret this repo touches is the gateway API key via `ApiKeySecretRef` above. -### Historian config (`ServerHistorian:Backend` — HistorianGateway or SQL Server) +### Historian config (`ServerHistorian:Backend` — HistorianGateway, SQL Server or QuestDB) The historian backend is **pluggable**, selected by `ServerHistorian:Backend` (case-insensitive; default `Gateway`; unknown value ⇒ host start fails listing the allowed values): `Gateway` is the external **`ZB.MOM.WW.HistorianGateway`** -sidecar below, `SqlServer` is a plain SQL Server database (see [`ServerHistorian:SqlServer`](#serverhistoriansqlserver-backendsqlserver)). +sidecar below, `SqlServer` is a plain SQL Server database (see [`ServerHistorian:SqlServer`](#serverhistoriansqlserver-backendsqlserver)), +and `QuestDb` is a QuestDB time-series database reached over the Postgres wire protocol (see +[`ServerHistorian:QuestDb`](#serverhistorianquestdb-backendquestdb)). The default backend is the gateway, consumed as the `ZB.MOM.WW.HistorianGateway.Client` gRPC package (the retired Wonderware TCP sidecar is documented at [`docs/drivers/Historian.Wonderware.md`](drivers/Historian.Wonderware.md)). The OtOpcUa host reads three @@ -404,13 +406,30 @@ Validated at host start (driver-role nodes) when `ServerHistorian:Enabled` **or* | Key | Default | Validation / meaning | |---|---|---| -| `ServerHistorian:Backend` | `Gateway` | `Gateway` or `SqlServer`; anything else fails host start (checked even when the historian is disabled). | +| `ServerHistorian:Backend` | `Gateway` | `Gateway`, `SqlServer` or `QuestDb`; anything else fails host start (checked even when the historian is disabled). | | `ServerHistorian:SqlServer:ConnectionString` | `""` | **Required.** Historian database connection string (not the ConfigDb). Supply via `${secret:OT-HISTORIAN-SQL}` or env `ServerHistorian__SqlServer__ConnectionString`; the database must already exist. | | `ServerHistorian:SqlServer:Schema` | `hist` | Must match `^[A-Za-z_][A-Za-z0-9_]{0,127}$` (spliced into DDL/DML). | | `ServerHistorian:SqlServer:CommandTimeout` | `00:00:30` | Must be `> 0`; rounded up to whole seconds. | | `ServerHistorian:SqlServer:AutoCreateSchema` | `true` | `true` applies the embedded schema script under an app-lock on first use; `false` only verifies `SchemaVersion >= 1` and fails every historian call (health `LastError`, no crash) if behind. | -HistoryUpdate (SqlServer only) additionally needs `ServerHistorian:Enabled=true` and a key whose principal holds the +#### `ServerHistorian:QuestDb` (`Backend=QuestDb`) + +Read only when `ServerHistorian:Backend` is `QuestDb`; the gateway keys are then not required. Validated at host start (driver-role nodes) +when `ServerHistorian:Enabled` **or** `AlarmHistorian:Enabled` is true (`QuestDbHistorianOptions.Validate()`). + +| Key | Default | Validation / meaning | +|---|---|---| +| `ServerHistorian:QuestDb:ConnectionString` | `""` | **Required.** Npgsql connection string for QuestDB's PG-wire port, e.g. `Host=questdb;Port=8812;Username=admin;Password=quest;Database=qdb;ServerCompatibilityMode=NoTypeLoading`. Env `ServerHistorian__QuestDb__ConnectionString` / `${secret:}` — never commit a real one. Set Npgsql `Maximum Pool Size` below QuestDB's 64-connection default limit (Npgsql's own default is 100), per node. | +| `ServerHistorian:QuestDb:TablePrefix` | `hist_` | Must match `^[A-Za-z0-9_]+$` (spliced into DDL/DML). Tables are `{prefix}sample`, `{prefix}event`, `{prefix}tag`. | +| `ServerHistorian:QuestDb:RetentionDays` | unset | `>= 1` when set. Applied as table TTL to the sample and event tables (enforced on write commits only — a quiet table never drops). Unset keeps data forever. | +| `ServerHistorian:QuestDb:CommandTimeout` | `00:00:30` | Must be `> 0`; rounded up to whole seconds. | +| `ServerHistorian:QuestDb:AutoCreateSchema` | `true` | `true` creates the tables and reconciles the TTL on first use; `false` only verifies the tables exist and fails every historian call (health `LastError`, no crash) if one is missing. | +| `ServerHistorian:QuestDb:VisibilityTimeout` | `00:00:05` | Must be `>= 0`. How long a write waits for its rows to become visible to queries (WAL apply); `0` disables the wait. A timeout only logs a Warning. | + +HistoryUpdate (soft delete) needs `ServerHistorian:Enabled=true` and a `HistoryUpdate` grant, as for `SqlServer`. Semantics, the µs timestamp rules, +pre-1970 limits, TTL, and the docker-dev overlay recipe: [`docs/Historian.md`](Historian.md#questdb-backend). + +HistoryUpdate (SqlServer and QuestDb only) additionally needs `ServerHistorian:Enabled=true` and a key whose principal holds the `HistoryUpdate` permission — see [`docs/security.md`](security.md#historyupdate-historian-writes). Full semantics, schema, aggregates and the docker-dev recipe: [`docs/Historian.md`](Historian.md#sql-server-backend). @@ -519,8 +538,9 @@ historian secret this repo reads from the environment is the gateway API key: | Variable | Effect / default | |---|---| | `ServerHistorian__ApiKey` | The HistorianGateway peppered-HMAC key (`histgw__`) sent as `Authorization: Bearer`. Supply via environment — **never commit**. Required when `ServerHistorian:Enabled=true` **and `Backend=Gateway`**. | -| `ServerHistorian__Backend` | `Gateway` (default) or `SqlServer`. | +| `ServerHistorian__Backend` | `Gateway` (default), `SqlServer` or `QuestDb`. | | `ServerHistorian__SqlServer__ConnectionString` | The SQL Server historian connection string. Required when `Backend=SqlServer` and the historian (or alarm history) is enabled. Supply via environment / `${secret:}` — **never commit**. | +| `ServerHistorian__QuestDb__ConnectionString` | The QuestDB historian Npgsql connection string. Required when `Backend=QuestDb` and the historian (or alarm history) is enabled. Supply via environment / `${secret:}` — **never commit**. | ### Driver integration-test / fixture sim endpoints diff --git a/docs/Historian.md b/docs/Historian.md index 2719e872..e18e8ae1 100644 --- a/docs/Historian.md +++ b/docs/Historian.md @@ -11,9 +11,10 @@ Wonderware TCP sidecar backend this replaced was retired — see [drivers/Historian.Wonderware.md](drivers/Historian.Wonderware.md).) > **The backend is pluggable (2026-10).** `ServerHistorian:Backend` selects `Gateway` (the default — everything -> described below as "the gateway") or `SqlServer` (a plain SQL Server database the server reads and writes -> itself, which also supports OPC UA **HistoryUpdate**). See [Backend selection](#backend-selection-serverhistorianbackend), -> [SQL Server backend](#sql-server-backend) and [HistoryUpdate](#historyupdate). Wherever this page says +> described below as "the gateway"), `SqlServer` (a plain SQL Server database the server reads and writes +> itself, which also supports OPC UA **HistoryUpdate**) or `QuestDb` (a QuestDB time-series database over the Postgres +> wire protocol; also supports HistoryUpdate, via soft delete). See [Backend selection](#backend-selection-serverhistorianbackend), +> [SQL Server backend](#sql-server-backend), [QuestDB backend](#questdb-backend) and [HistoryUpdate](#historyupdate). Wherever this page says > "the historian" without qualification it means the selected backend; sections that only apply to one backend say so. Design references: [docs/plans/2026-06-14-galaxy-phase-c-historian-design.md](plans/2026-06-14-galaxy-phase-c-historian-design.md) @@ -66,23 +67,23 @@ A tag that is both historized and a native alarm: (`IHistorianBackendProvider`) that builds all five historian seams — the HistoryRead data source, the continuous- historization value writer, the alarm-history writer, the tag provisioner and (optionally) the HistoryUpdate updater. `HistorianBackendRegistry` in the Host owns the name table. An unknown name fails host start with an -`OptionsValidationException` that lists the allowed values (`Gateway`, `SqlServer`), regardless of whether anything +`OptionsValidationException` that lists the allowed values (`Gateway`, `SqlServer`, `QuestDb`), regardless of whether anything consumes the historian. The existing enable gates (`ServerHistorian:Enabled`, `AlarmHistorian:Enabled`, `ContinuousHistorization:Enabled`) are unchanged — they simply get their objects from the selected provider. -| | `Gateway` (default) | `SqlServer` | -|---|---|---| -| Storage | AVEVA Historian via the HistorianGateway sidecar | `{Schema}` tables in a SQL Server database you point it at | -| Value kinds recorded | `Double`, `Int64`, `Boolean` (numeric analog only) | all: `Double`, `Int64`, `Boolean`, `String`, `DateTime`, plus `Empty` (a Bad sample with no value) | -| Status code stored | Sent as DA quality but **discarded** by the gateway (C-002); AVEVA stores every sample Good | Stored bit-for-bit (`StatusCode` int column) | -| Comms-loss / driver-Bad publishes | Dropped, counted in `DroppedUnsupportedKind` | Stored as an `Empty` outage-marker sample carrying the Bad code (see [below](#typed-values-real-status-and-outage-markers)) | -| Processed aggregates | Average, Minimum, Maximum, Total, Count | The thirteen in the [aggregate table](#aggregates-processed-reads) | -| HistoryUpdate | No (`BadHistoryOperationUnsupported`) | Yes (Insert / Replace / Update / DeleteRaw / DeleteAtTime) | -| Alarm history | gateway `SendEvent` | `Event` table (insert-if-absent) | -| Provisioning (`EnsureTags`) | numeric types only; others skipped | `MERGE` upsert of every request, all types | -| Connection | `Endpoint` + `ApiKey` + TLS keys (top-level `ServerHistorian:*`) | `ServerHistorian:SqlServer:*` | +| | `Gateway` (default) | `SqlServer` | `QuestDb` | +|---|---|---|---| +| Storage | AVEVA Historian via the HistorianGateway sidecar | `{Schema}` tables in a SQL Server database you point it at | `{prefix}sample` / `event` / `tag` WAL tables in a QuestDB instance | +| Value kinds recorded | `Double`, `Int64`, `Boolean` (numeric analog only) | all: `Double`, `Int64`, `Boolean`, `String`, `DateTime`, plus `Empty` (a Bad sample with no value) | same as `SqlServer` | +| Status code stored | Sent as DA quality but **discarded** by the gateway (C-002); AVEVA stores every sample Good | Stored bit-for-bit (`StatusCode` int column) | Stored bit-for-bit (unsigned, in a `LONG`) | +| Comms-loss / driver-Bad publishes | Dropped, counted in `DroppedUnsupportedKind` | Stored as an `Empty` outage-marker sample carrying the Bad code (see [below](#typed-values-real-status-and-outage-markers)) | Stored as an `Empty` outage-marker sample, as on `SqlServer` | +| Processed aggregates | Average, Minimum, Maximum, Total, Count | The thirteen in the [aggregate table](#aggregates-processed-reads) | The thirteen, pushed down to QuestDB SQL | +| HistoryUpdate | No (`BadHistoryOperationUnsupported`) | Yes (Insert / Replace / Update / DeleteRaw / DeleteAtTime) | Yes — soft delete (tombstones), see [below](#historyupdate-soft-delete) | +| Alarm history | gateway `SendEvent` | `Event` table (insert-if-absent) | `{prefix}event` table (dedup upsert on `(ts, event_id)`) | +| Provisioning (`EnsureTags`) | numeric types only; others skipped | `MERGE` upsert of every request, all types | Upsert into `{prefix}tag`, all types | +| Connection | `Endpoint` + `ApiKey` + TLS keys (top-level `ServerHistorian:*`) | `ServerHistorian:SqlServer:*` | `ServerHistorian:QuestDb:*` | -Recording stays **Primary-only** (#537) and **at-least-once** with the #538 per-tag quarantine on both backends; +Recording stays **Primary-only** (#537) and **at-least-once** with the #538 per-tag quarantine on every backend; HistoryRead and HistoryUpdate are served from any node (the database is shared). The historian connection string is **not** a ConfigDb string, and a driver-only node may carry it without breaking the Phase 4 "no ConfigDb" posture. @@ -114,6 +115,14 @@ and all HistoryRead calls on historized nodes return `GoodNoData` (empty, not an "Schema": "hist", "CommandTimeout": "00:00:30", "AutoCreateSchema": true + }, + "QuestDb": { + "ConnectionString": "", + "TablePrefix": "hist_", + "RetentionDays": null, + "CommandTimeout": "00:00:30", + "AutoCreateSchema": true, + "VisibilityTimeout": "00:00:05" } } } @@ -122,7 +131,7 @@ and all HistoryRead calls on historized nodes return `GoodNoData` (empty, not an | Key | Type | Default | Description | |---|---|---|---| | `Enabled` | bool | `false` | Enable the selected backend's read client. `false` → `NullHistorianDataSource` (empty reads) and no HistoryUpdate. | -| `Backend` | string | `Gateway` | `Gateway` or `SqlServer` (case-insensitive). Unknown ⇒ host start fails. See [Backend selection](#backend-selection-serverhistorianbackend). | +| `Backend` | string | `Gateway` | `Gateway`, `SqlServer` or `QuestDb` (case-insensitive). Unknown ⇒ host start fails. See [Backend selection](#backend-selection-serverhistorianbackend). | | `Endpoint` | string | `""` | **Gateway backend only.** Absolute gateway URI, e.g. `https://host:5222`. Scheme selects transport (`https://` = TLS, `http://` = h2c plaintext). Required when `Enabled`. | | `ApiKey` | string | `""` | **Gateway backend only.** The gateway peppered-HMAC key (`histgw__`) sent as `Authorization: Bearer`. Required when `Enabled`. **Supply via env `ServerHistorian__ApiKey`.** | | `UseTls` | bool | `true` | **Gateway backend only.** Connect over TLS; must match the `Endpoint` scheme. | @@ -143,6 +152,10 @@ and all HistoryRead calls on historized nodes return `GoodNoData` (empty, not an | `SqlServer:CommandTimeout` | TimeSpan | `00:00:30` | Must be `> 0` | Per-command timeout, rounded up to whole seconds (minimum 1 s). The SQL backend does **not** use `CallTimeout`. | | `SqlServer:AutoCreateSchema` | bool | `true` | — | `true`: on first use apply any missing embedded script under a `sp_getapplock` (`OtOpcUaHistorianSchema`), so concurrent nodes serialise. `false`: only verify `SchemaVersion >= 1`; if behind, **every** historian call fails with a clear error naming the missing version (surfaced in the read client's health `LastError`, never a host crash). A failed bootstrap is retried on the next call, not latched. | +**`ServerHistorian:QuestDb`** (read only when `Backend=QuestDb`; bound by `HistorianBackendRegistry.BindQuestDb`): six keys — `ConnectionString` +(required), `TablePrefix`, `RetentionDays`, `CommandTimeout`, `AutoCreateSchema`, `VisibilityTimeout`. The full table, defaults and validation +rules are in [QuestDB backend → Configuration](#configuration-serverhistorianquestdb) (and [Configuration.md](Configuration.md#serverhistorianquestdb-backendquestdb)). + **Validation rules** (`ServerHistorianOptionsValidator`, driver-role nodes only; admin-only nodes never build historian singletons): - `Backend` must name a registered backend — checked **always**, with no I/O, even when nothing is enabled. @@ -150,6 +163,9 @@ and all HistoryRead calls on historized nodes return `GoodNoData` (empty, not an (the alarm sink sources its connection from this section even when `ServerHistorian` is off). - `Backend=Gateway`: `Endpoint` must be an absolute http(s) URI, `ApiKey` non-empty, and `UseTls` must agree with the endpoint scheme. - `Backend=SqlServer`: the three `SqlServer` rules above. `Endpoint`/`ApiKey`/`UseTls` are **not** required. + - `Backend=QuestDb`: `QuestDbHistorianOptions.Validate()` (connection string required, prefix regex, `RetentionDays >= 1`, `CommandTimeout > 0`, + `VisibilityTimeout >= 0`). `Endpoint`/`ApiKey`/`UseTls` are **not** required. A malformed `TablePrefix` is also rejected by the connection helper + when the provider first builds a seam. - ⚠️ `Backend=SqlServer` with a malformed `Schema` throws `ArgumentException` while the provider is built, so it stops host start even when no consumer is enabled (the validator is consumer-gated, provider construction is not). - The alarm-only warning at startup (`AlarmHistorian` on, `ServerHistorian` off) names the selected backend. @@ -261,7 +277,7 @@ by `SqlServer`, not by `Gateway`). (#481) fans `(null, BadNoCommunication)` for every value ref of a comms-lost driver to the dependency mux, and the recorder cannot tell that from a driver's own Bad publish. So: -- **`SqlServer`:** each disconnect stores **one `Empty` / `BadNoCommunication` (`0x80310000`) sample per historized +- **`SqlServer` and `QuestDb`:** each disconnect stores **one `Empty` / `BadNoCommunication` (`0x80310000`) sample per historized tag**. A driver's own Bad publish is stored the same way (`Empty` kind plus its own code — e.g. Modbus `BadCommunicationError` `0x80050000`). This is deliberate: raw reads show the gap, and aggregates and AtTime interpolation do not bridge it. Cost: one extra row per tag per outage. @@ -288,7 +304,7 @@ rolling a node back, let the outbox drain to zero (`otopcua.historization.outbox delete the directory named by `ContinuousHistorization:OutboxPath` (accepting the loss of anything un-drained). Upgrading needs no action. -**Delivery semantics on `SqlServer`.** Writes are *insert-if-absent* on `(TagId, SourceTimeUtc)`: the outbox's +**Delivery semantics on `SqlServer`** (`QuestDb` is the same, at µs resolution and with tombstones counting as present — see [QuestDB backend](#live-writes-insert-if-absent-tombstones-included)). Writes are *insert-if-absent* on `(TagId, SourceTimeUtc)`: the outbox's at-least-once redelivery (and both pair nodes writing the same sample in a boot window) converges on one row, and the first-stored sample wins. **Duplicate timestamps inside one batch collapse to the first.** A `NaN` / `±Infinity` `Double` cannot be stored in a SQL `float`, so the **value writer stores it as an `Empty` sample** — the tag is not quarantined for @@ -797,10 +813,188 @@ dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.Integrati --- +## QuestDB backend + +Selected with `ServerHistorian:Backend=QuestDb`. Implemented in `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb` +(pure logic shared with the SQL Server backend lives in `Driver.Historian.Sql.Common`). All access is the **Postgres wire protocol via +`Npgsql`** (no ILP, no extra package). Design: [plans/2026-10-10-questdb-historian-design.md](plans/2026-10-10-questdb-historian-design.md); +the spike that pinned every QuestDB-specific form: [plans/2026-10-10-questdb-spike.md](plans/2026-10-10-questdb-spike.md). Pinned image: +`questdb/questdb:10.0.1`. + +QuestDB serves every value kind (`Double`, `Int64`, `Boolean`, `String`, `DateTime`, `Empty`), stores the real status, records comms-loss +outage markers, serves the thirteen aggregates and supports **HistoryUpdate** (soft delete), exactly like `SqlServer`. The same conformance +suite (`tests/Drivers/Historian.Conformance`) runs against both backends; the differences below are the only ones. + +### Configuration (`ServerHistorian:QuestDb`) + +Read only when `Backend=QuestDb`; bound by `HistorianBackendRegistry.BindQuestDb`, validated (consumer-gated, like `SqlServer`) by +`QuestDbHistorianOptions.Validate()`. The gateway keys (`Endpoint`, `ApiKey`, `UseTls`) are not required. + +| Key | Default | Validation | Meaning | +|---|---|---|---| +| `QuestDb:ConnectionString` | `""` | **Required** (non-blank) | Npgsql connection string for QuestDB's PG-wire port, e.g. `Host=questdb;Port=8812;Username=admin;Password=quest;Database=qdb;ServerCompatibilityMode=NoTypeLoading`. `ServerCompatibilityMode=NoTypeLoading` skips the `pg_type` load; every query form was verified with it. Supply via env `ServerHistorian__QuestDb__ConnectionString` / `${secret:}` for anything real. | +| `QuestDb:TablePrefix` | `hist_` | `^[A-Za-z0-9_]+$` | QuestDB has no schemas, so the three tables are `{prefix}sample`, `{prefix}event`, `{prefix}tag`. Spliced into DDL/DML, hence the regex. | +| `QuestDb:RetentionDays` | unset | `>= 1` when set | Applied as table TTL to `{prefix}sample` and `{prefix}event` (see [Retention](#retention-ttl)). Unset keeps data forever. | +| `QuestDb:CommandTimeout` | `00:00:30` | `> 0` | Per-command timeout, rounded up to whole seconds (minimum 1 s). | +| `QuestDb:AutoCreateSchema` | `true` | — | `true`: create the tables (and reconcile the TTL) on first use. `false`: only verify the three tables exist; if one is missing every historian call fails with `HistorianSchemaMissingException` (health `LastError`, no crash) and no TTL is touched. A failed bootstrap is not latched: it is re-thrown for 5 s without touching QuestDB, then retried. | +| `QuestDb:VisibilityTimeout` | `00:00:05` | `>= 0` | How long a write waits for its rows to become visible to queries ([WAL visibility](#wal-visibility-and-the-read-your-writes-wait)). `0` disables the wait. | + +⚠️ **Connection pool size.** QuestDB's default limit is **64** concurrent PG-wire connections (`pg.net.connection.limit`), while Npgsql's +default `Maximum Pool Size` is **100**. A busy node (HistoryRead batches, the recorder, the updater and alarm writes all share one pool) +can therefore be refused by QuestDB before Npgsql's own pool is full. Set `Maximum Pool Size` in the connection string below the server +limit — and below it **per node**, since a redundant pair shares one QuestDB: for example `…;Maximum Pool Size=24` on each of two nodes. + +### Tables + +All three are **WAL tables with `DEDUP UPSERT KEYS`**, so re-sending a row with the same key replaces it (the upsert is what makes +HistoryUpdate Replace work; the writer's own insert-if-absent semantics are built on top, below). + +| Table | Key | Partitioning | +|---|---|---| +| `{p}sample` (`ts, tag, kind, num_value, int_value, str_value, status, deleted, server_ts`) | `(ts, tag)` | `PARTITION BY DAY` (TTL when configured) | +| `{p}event` (one-for-one with `hist.Event`) | `(ts, event_id)` | `PARTITION BY DAY` (TTL when configured) | +| `{p}tag` (`ts, name, data_type, eng_unit, description, updated_utc`) | `(ts, name)` with `ts` fixed at `1970-01-01` — an upsert by name | `PARTITION BY YEAR`, never expires | + +`status` holds the uint OPC UA status code **unsigned** in a `LONG` (so "Bad" is `status >= 2147483648`, not `< 0`); `int_value` also holds +Booleans as 0/1; `kind` is the `HistorianValueKind` in a `BYTE`. Tag names are limited to 400 characters (longer is a per-tag failure, +never truncated). `{p}tag` is the **only** "is this tag known" authority (`QuestDbTagCatalog`): reads never create a tag, Insert/Update and the +live writer do, Replace and the deletes do not. + +### Timestamps: microseconds + +QuestDB's designated timestamp is a microsecond. Every inbound `DateTime` is normalised by `QuestDbConnection.Round` (Unspecified ⇒ UTC, Local +⇒ UTC, ticks truncated to a multiple of 10, `MinValue`/`MaxValue` nudged inward because Npgsql would send them as ±infinity) and bound only +through `BindTimestamp`. Consequences: + +- Two live samples for one tag **less than 1 µs apart** collapse to one; **the first wins**. A batch is deduplicated client-side after rounding. +- **Raw and Processed results carry stored (µs) timestamps.** A sub-µs request bound is rounded, and the rounded start *is* the request start, + so a Processed bucket's timestamp is the rounded window start (a sub-µs label divergence from `SqlServer`, which labels with the exact + start; values and bucket membership are identical). +- **AtTime** evaluates at the rounded time but **stamps each result with the time the client asked for**, so a client can match answers to + requests. Stored data is µs-only, so a sub-µs request adjacent to a sample is an exact hit. +- `IHistorianDataSource.TimestampResolutionTicks` is **1** by default and **10** on QuestDB. HistoryRead-Raw paging steps off a drained + timestamp by that amount (forward `T + resolution`, reverse `T − resolution`). Stepping by 1 tick would be truncated back to `T` and a + one-value-per-page forward read would loop forever. Same-timestamp ties (legal on the Gateway backend) are still paged within the timestamp. +- **Pre-1970 timestamps cannot be stored** (QuestDB: `designated timestamp before 1970-01-01 is not allowed`; the whole INSERT fails). So: + - the **live writer** drops such samples with one Warning per batch (counted in `DroppedPre1970`) — a retry could never succeed, and sending + them would push the tag into the #538 quarantine; + - **HistoryUpdate** Insert / Replace / Update of such an item returns `BadOutOfRange` (nothing is sent); `DeleteAtTime` returns + `BadNoEntryExists`; `DeleteRaw` finds nothing; + - an alarm event with such a time is a `PermanentFail` (dead-letter); + - **read bounds before 1970 work** (a Raw/Processed window may start before the epoch; bounds are never clamped, only `MinValue`/`MaxValue` nudged). +- **`Int64` `long.MinValue`.** QuestDB's `LONG` NULL is `long.MinValue`, so a stored `FromInt64(long.MinValue)` has `int_value` NULL. The read path + repairs it (`kind = Int64 ∧ int_value IS NULL ∧ num_value = −2^63` ⇒ `long.MinValue`); without the repair it would read back as `Empty`/0. Likewise + `NaN`/`±Infinity` doubles read back as NULL, so they are stored as `Empty` + `BadOutOfRange` (live) or rejected `BadInvalidArgument` (HistoryUpdate), as on `SqlServer`. + +### Live writes: insert-if-absent, tombstones included + +The recorder's writer (`QuestDbHistorianValueWriter`) has the same first-wins redelivery contract as SQL Server: + +1. group by tag, round to µs, drop in-batch duplicates (first wins), drop pre-1970 samples; +2. read the timestamps that already exist for the tag (one range query per cluster of nearby times, **`deleted` rows included**); +3. `INSERT` only the absent rows — parameterised multi-row, at most **1,000 rows per statement**; +4. [wait for WAL visibility](#wal-visibility-and-the-read-your-writes-wait). + +Because `DEDUP UPSERT` would otherwise let a re-sent sample overwrite the stored one, step 2 is what keeps outbox redelivery idempotent — and +**a live-writer redelivery never resurrects a deleted row**: a tombstone counts as present. Any SQL or transport failure returns `false` +(the recorder retries; the #538 per-tag quarantine still applies); the writer never throws. + +### WAL visibility and the read-your-writes wait + +QuestDB applies WAL commits asynchronously, so a row just inserted may not be visible to the next query (measured: up to 7 of 20 inserts invisible +to an immediate re-read). **Every write path waits** — the live writer, the HistoryUpdate updater (including between DeleteRaw pages), the tag +catalog/provisioner and the alarm writer (`QuestDbWalVisibility`): after the INSERT(s) it reads `sequencerTxn` from `wal_tables()` and polls +`writerTxn` about every 10 ms until it catches up, the table is reported `suspended` (Warning, stop), or `VisibilityTimeout` elapses (Warning). +A timeout never changes a result code and never discards data; the rows simply become visible shortly after. Measured: median 0 ms, p95 6 ms. + +### HistoryUpdate (soft delete) + +All five operations are supported, and **every one is an `INSERT`**: QuestDB `UPDATE`/`DELETE` are never issued. "Exists" means a row that is +present **and not deleted**. + +| Operation | Effect | Result | +|---|---|---| +| Insert | write the row when none exists — **a tombstone counts as absent, so Insert over a tombstone succeeds** | `GoodEntryInserted` / `BadEntryExists` | +| Replace | overwrite (dedup upsert) when a live row exists | `GoodEntryReplaced` / `BadNoEntryExists` | +| Update | write either way | `GoodEntryReplaced` or `GoodEntryInserted` | +| DeleteAtTime | write a tombstone (`deleted = true`, value NULL, `BadNoData`) where a live row exists | `Good` / `BadNoEntryExists` | +| DeleteRaw | tombstone every live row in `[start, end)`, 1,000 per page, waiting for visibility between pages | the same codes as `SqlServer` | + +The per-item codes come from the shared `HistoryUpdatePlanner`, so they equal `SqlServer`'s (including the duplicate-timestamp-in-one-request rules +above). **Every read and aggregate filters `NOT deleted`.** A client Insert at a deleted timestamp re-populates it (the OPC UA intent); only the +*live writer's* redelivery is barred from doing so. + +⚠️ **Concurrency.** The updater serializes calls **per tag, in-process only** (64 striped locks held from the existence read through the visibility +wait), so two concurrent Insert/Replace/Delete batches on one tag cannot both act on a stale read. It does **not** lock against the live writer, +nor against a second node. The existence checks read committed data, so a recorder sample at the same timestamp still in flight in the WAL can race +an Insert/Replace (updates normally target past timestamps), and two nodes writing the same *new* key concurrently can produce last-write-wins +(QuestDB has no `UPDLOCK`; the Primary gate makes it rare). This residual is accepted. + +### Reads + +Every query filters `NOT deleted`; every value goes through `StoredSample`, so the quality clamp equals `SqlServer`'s. + +- **Raw:** `ts` range, `ORDER BY ts [DESC] LIMIT $n` (bound, `int`); `ContinuationPoint` is always null — the node manager pages by re-reading + from the last timestamp ± `TimestampResolutionTicks`. +- **AtTime:** previous-or-equal and next neighbour of each time by `LIMIT 1` seeks, `UNION ALL`-ed, **50 times per statement** (QuestDB 10.0.1 refuses + more than 126 branches). `AtTimeEvaluator` decides each value, as on `SqlServer`. +- **Events:** `[start, end)` by `ts`, ordered `(ts, event_id)`, `maxEvents + 1` fetched to produce the continuation point. +- **Processed:** one query per node; bucket index `(cast(ts AS LONG) * 10 - startUnixTicks) / intervalTicks` (never `SAMPLE BY`, whose alignment + differs), so bucket edges are identical to `SqlServer` (start-inclusive, end-exclusive, partial last bucket, interval 0 ⇒ one bucket, 100,000-bucket cap). + Min/max times use `row_number()` windows with the earliest-timestamp tie-break. + +### Retention (TTL) + +`RetentionDays` becomes `TTL n DAYS` on `{p}sample` and `{p}event` (never on `{p}tag`: TTL must be a multiple of the partition size). QuestDB drops +expired day partitions itself, tombstones included. + +- At bootstrap the current TTL is read from `tables()` (`ttlValue`, `ttlUnit`). QuestDB **normalises the unit** (7 days reads `1 WEEK`), so the + bootstrapper compares in **hours** (`HOUR`=1, `DAY`=24, `WEEK`=168; any other unit counts as different) and issues + `ALTER TABLE … SET TTL n DAYS` only on a difference (`SET TTL 0 DAYS` removes it). The `ALTER` is applied through the WAL, so it does not + re-read to verify; a very large `RetentionDays` that QuestDB normalises into MONTH/YEAR is re-`ALTER`ed on every boot (harmless). +- ⚠️ **TTL is enforced only on write commits.** QuestDB checks for expired partitions when a write commits to the table, so **a quiet table never + drops anything**: partitions older than the TTL linger until the next write lands. Do not read a stale-looking partition as a broken TTL. +- Expiry is judged against the wall clock, so a future-dated sample does not expire current data. +- The TTL is visible in table metadata: `SELECT table_name, ttlValue, ttlUnit FROM tables() WHERE starts_with(table_name, 'hist_')`. + +### Docker-dev recipe + +QuestDB is an **optional compose overlay**, `docker-dev/docker-compose.questdb.yml`, so the default rig is unchanged. See +[`docker-dev/README.md`](../docker-dev/README.md#questdb-historian-overlay). + +```bash +# 1. Bring up ONLY the questdb service (PG wire on host :18812, console/REST on host :19000). +docker compose -f docker-dev/docker-compose.yml -f docker-dev/docker-compose.questdb.yml up -d --no-deps questdb + +# 2. Recreate the central nodes against it (the QuestDb connection string already defaults to the `questdb` service): +OTOPCUA_HISTORIAN_ENABLED=true OTOPCUA_HISTORIAN_BACKEND=QuestDb OTOPCUA_CONTHIST_ENABLED=true \ + docker compose -f docker-dev/docker-compose.yml -f docker-dev/docker-compose.questdb.yml up -d central-1 central-2 + +# 3. Mark a rig tag historized, deploy, then watch rows land (REST, no client needed): +curl -G localhost:19000/exec --data-urlencode "query=SELECT tag, count() n, max(ts) last FROM hist_sample WHERE NOT deleted GROUP BY tag" +``` + +Only the Primary of a pair records. Read back with `Client.CLI historyread`, rewrite with `Client.CLI historyupdate`. + +### Integration tests + +`tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests` (bootstrap incl. two concurrent bootstrappers, TTL via `tables()`, writes, +tombstones and redelivery, the visibility wait, updater, reads, paging, and the shared conformance suite) is gated on +**`HISTORIAN_QUESTDB_TEST_CONNECTION`** — an Npgsql connection string for the PG-wire port. Absent ⇒ every test skips. Each test class uses a +unique throwaway prefix (`t__`) and drops its tables afterwards. The QuestDB server allows 64 connections, so the fixture prunes idle pool connections. + +```bash +docker compose -f docker-dev/docker-compose.yml -f docker-dev/docker-compose.questdb.yml up -d --no-deps questdb +export HISTORIAN_QUESTDB_TEST_CONNECTION='Host=localhost;Port=18812;Username=admin;Password=quest;Database=qdb;ServerCompatibilityMode=NoTypeLoading' +dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests +``` + +--- + ## HistoryUpdate OPC UA **HistoryUpdate** lets a client insert, replace, update and delete stored raw history. It is available only when the selected -backend provides an updater — **`SqlServer` does; `Gateway` does not**, and an updater is registered only when `ServerHistorian:Enabled=true`. +backend provides an updater — **`SqlServer` and `QuestDb` do; `Gateway` does not**, and an updater is registered only when `ServerHistorian:Enabled=true`. Updates go straight to the backend (not through the outbox), are **not Primary-gated** (the database is shared), and share the HistoryRead batch limiter and `HistoryReadDeadline`. diff --git a/docs/plans/2026-10-10-questdb-historian-design.md b/docs/plans/2026-10-10-questdb-historian-design.md index c2f1ec99..2e6a3ba2 100644 --- a/docs/plans/2026-10-10-questdb-historian-design.md +++ b/docs/plans/2026-10-10-questdb-historian-design.md @@ -1,6 +1,7 @@ # QuestDB historian backend — design -**Status:** approved section by section on 2026-10-10. Gitea #666. +**Status:** approved section by section on 2026-10-10. Gitea #666. **As-built pass (T8, 2026-10-11):** every section below was re-checked against +the merged code; where the two disagreed the text was corrected inline and marked **(as built)**. User documentation: `docs/Historian.md` §"QuestDB backend". **Builds on:** `docs/plans/2026-10-09-pluggable-historian-design.md`, which defines the provider seam, typed values, the HistoryUpdate service and the SQL Server backend, all merged to master as `8ec7a83c`. @@ -12,7 +13,7 @@ HistoryUpdate service and the SQL Server backend, all merged to master as `8ec7a | Write path | **Postgres wire protocol only, via `Npgsql`.** It is already centrally pinned at 10.0.3. No ILP, no new NuGet package. | | Retention | **Included.** `RetentionDays` is applied as table TTL. QuestDB drops expired day partitions itself. | | Approach | **B: shared dialect-free core plus per-backend SQL.** The pure logic moves out of the SQL Server backend into `Driver.Historian.Sql.Common`. One conformance suite runs against both backends. | -| Timestamps | **Microseconds.** Postgres-wire timestamps are microsecond resolution, and real source timestamps are at best milliseconds. Two samples less than 1 µs apart within a batch collapse to one, and the first wins. This is documented. | +| Timestamps | **Microseconds.** Postgres-wire timestamps are microsecond resolution, and real source timestamps are at best milliseconds. Two samples less than 1 µs apart collapse to one, and the first wins — within a batch, and also against an already-stored row (the existence check runs on the rounded time). This is documented. | Out of scope: - event HistoryUpdate; @@ -62,8 +63,12 @@ seam classes, all calling the shared core. | `QuestDbConnection` | Npgsql `NpgsqlDataSource` (pooled), command timeouts, transient-error classification, UTC timestamp reads. | | `QuestDbSchemaBootstrapper` | Idempotent `CREATE TABLE IF NOT EXISTS` for the three tables, then applies TTL. It is concurrency-safe across two nodes starting together, and caches a failure for 5 s. | | `QuestDbHistorianValueWriter` | `IHistorianValueWriter` for live samples (§2.2). | -| `QuestDbTagProvisioner` | `IHistorianProvisioning`: upserts `hist_tag` rows. | -| `QuestDbAlarmHistorianWriter` | `IAlarmHistorianWriter`: writes events with the deterministic SHA-256 `event_id` and the same classify rules as SQL Server (unknown ⇒ RetryPlease). | +| `QuestDbTagCatalog` | **(as built)** The only "is this tag known" authority (R7): a `hist_tag` row, positive results cached. Reads never create a tag; the live writer and HistoryUpdate Insert/Update do (Replace and the deletes do not). Re-inserting a known tag is skipped, because the dedup upsert would erase the unit and description the provisioner wrote. | +| `QuestDbWalVisibility` | **(as built)** The read-your-writes wait (§2.3), used by every write path. | +| `QuestDbTimeClusters` | **(as built)** Groups request times into ranges (gap > 1 min starts a new one) so existence reads are one range query per cluster. | +| `QuestDbTables`, `QuestDbAggregateQueries` | **(as built)** DDL / TTL helpers, and the push-down aggregate SQL (§3). | +| `QuestDbTagProvisioner` | `IHistorianProvisioning`: upserts `hist_tag` rows in chunks; **(as built)** a request without a unit or description keeps the stored one. | +| `QuestDbAlarmHistorianWriter` | `IAlarmHistorianWriter`: **(as built)** one INSERT per event (not batched), with the deterministic SHA-256 `event_id`. Outcomes: Ack; `RetryPlease` for transient and unclassified errors; `PermanentFail` only for a pre-1970 time or a server message naming an inconvertible / out-of-range / too-long / null value. | | `QuestDbHistorianDataSource` | `IHistorianDataSource` (§3), with QuestDB SQL for raw, at-time, events and bucket statistics. | | `QuestDbHistorianDataUpdater` | `IHistorianDataUpdater` (§2.3), the soft-delete updater. | | `QuestDbHistorianProvider` | `IHistorianBackendProvider`: `Name = "QuestDb"`, `SupportedKinds` = all six (FrozenSet), one shared connection and bootstrapper. | @@ -73,6 +78,8 @@ Host changes: - `ServerHistorianOptionsValidator` validates the QuestDb section, gated on consumers, the same way it gates SqlServer. - `Host.csproj` gets a ProjectReference. +- **(as built)** `IHistorianDataSource` gains the default member `TimestampResolutionTicks` (1; QuestDB 10), and the node manager's Raw paging steps off a drained timestamp by it (§3). +- **(as built)** The provider builds its shared pieces (connection, bootstrapper, catalog, WAL wait) on the first `Create*` call, not in the constructor, so a provider nobody consumes never fails on its options. ## 2. Schema and writes @@ -132,11 +139,11 @@ Column details (**T0-confirmed**): - `long.MinValue` (LONG) and `int.MinValue` (INT) are QuestDB's NULL sentinels. - `BYTE` and `BOOLEAN` have **no NULL**: unset or DBNull reads `0` / `false`. So `deleted` defaults to `false`, and a LEFT-JOINed "no row" yields `kind = 0`. Absence must be detected on a NULL `ts`, not on `kind`. -- **Ruling candidate (T0):** an Int64 sample of exactly `long.MinValue` stores `int_value` as NULL. The QuestDB read +- **Ruling RC2 (as built):** an Int64 sample of exactly `long.MinValue` stores `int_value` as NULL. The QuestDB read path repairs it before the shared `FromColumns`: `kind = Int64 ∧ int_value IS NULL ∧ num_value = -2^63` ⇒ `long.MinValue`. That double is exact. Without the repair the sample would be served as 0. - A designated timestamp **before 1970-01-01 cannot be inserted** (`designated timestamp before 1970-01-01 is not - allowed`). See §2.2. + allowed`). See §2.2 (RC1). - Tag names: SYMBOL and VARCHAR take 400 characters (non-ASCII included). SYMBOL grows past its capacity hint (16,384) with no error, and was tested with 30,000 distinct tags. No `CAPACITY` clause is needed. @@ -144,20 +151,24 @@ Column details (**T0-confirmed**): 1. Group the batch by tag and remove duplicates within the batch. **The first value wins.** A timestamp is rounded to microseconds before this step, so microsecond collisions also collapse. -2. For each tag, fetch the timestamps that already exist in the batch's `[min ts, max ts]` range. **Tombstones are - included**, so a re-sent outbox batch can never resurrect a deleted row or overwrite a HistoryUpdate change. - T0-confirmed form (Q11): `SELECT ts FROM {p}sample WHERE tag = $1 AND ts >= $2 AND ts <= $3`, with no `NOT - deleted`, followed by a client-side set test. +2. For each tag, fetch the timestamps that already exist. **(as built)** The batch's times are grouped into clusters + (`QuestDbTimeClusters`: a gap over 1 minute starts a new range; the live writer also caps a range at 1,000 times), with one range query per cluster, so one + far-away outlier does not widen a single query over every stored row in between. **Tombstones are included**, so a re-sent + outbox batch can never resurrect a deleted row or overwrite a HistoryUpdate change. T0-confirmed form (Q11): `SELECT ts + FROM {p}sample WHERE tag = $1 AND ts >= $2 AND ts <= $3`, with no `NOT deleted`, followed by a client-side set test. The tag is + registered in `QuestDbTagCatalog` first (a write creates an unknown tag). 3. Insert only the new rows with parameterised multi-row `INSERT … VALUES (…),(…)`, **at most 1,000 rows per statement** (T0-confirmed Q2). QuestDB reads the Bind parameter count as a signed int16, so the hard limit is - 32,767 parameters (3,640 nine-column rows). Within one statement a duplicate key keeps the **last** row, which is + 32,767 parameters. **(as built)** The live writer binds 8 parameters per row (`deleted` is the literal `false`; 4,095 rows + would fit), the updater 9 (3,640 rows). Within one statement a duplicate key keeps the **last** row, which is why step 1's "first wins" is done client-side. 4. Stored-value rules are shared with SQL Server: - NaN/±Inf, or any Empty with a non-Bad status, is stored Empty with `BadOutOfRange` or `BadNoData`. - Comms-loss outage markers are stored as given. - A tag name over 400 characters is a per-tag failure. - - **Ruling candidate (T0 Q13):** a sample timestamped before 1970-01-01 cannot be stored, because QuestDB rejects - the whole INSERT. The writer drops such samples before the INSERT and logs one Warning per batch. A retry could + - **Ruling RC1 (T0 Q13):** a sample timestamped before 1970-01-01 cannot be stored, because QuestDB rejects + the whole INSERT. The writer drops such samples before the INSERT, logs one Warning per batch and counts them + (`DroppedPre1970`); the call still succeeds for the rest. A retry could never succeed, and sending the sample would push the tag's whole group into the #538 quarantine. 5. Any SQL or transport failure makes the write return `false`, so the recorder retries and the #538 per-tag quarantine still applies. It never throws. @@ -166,7 +177,10 @@ The recorder is Primary-only (#537), so step 2 does not race another live writer visibility, bounded by `VisibilityTimeout` (plan R5), using the §2.3 wait. Binding (T0-confirmed Q1/Q6/Q13): -- Every timestamp parameter is `NpgsqlDbType.TimestampTz` with `DateTimeKind.Utc`, after `Round`. `Timestamp` + `Utc` +- Every timestamp parameter is `NpgsqlDbType.TimestampTz` with `DateTimeKind.Utc`, after `Round` (**as built:** Unspecified is + taken as UTC, Local is converted, then ticks are truncated to a multiple of 10). `BindTimestamp(…, clampToEpoch: true)` is used + only for write binds of a designated timestamp; read bounds are never clamped (a pre-1970 bound is valid, and clamping it would + move the bound across the epoch row). `Timestamp` + `Utc` throws in Npgsql. Npgsql truncates sub-µs ticks **toward 2000-01-01**, which rounds *up* before 2000. A raw `DateTime.MinValue`/`MaxValue` is sent as ±infinity, and a `[Min, Max)` range then silently matches nothing. R6's `BindTimestamp` (round; `MinValue` → `+10` ticks; `MaxValue` → `−9` ticks) fixes all three. @@ -193,7 +207,7 @@ issued. "Exists" means a row that is present **and not deleted**. | Operation | Effect | Result | |---|---|---| -| Insert | Write the row when none exists; a tombstone counts as absent | `GoodEntryInserted` / `BadEntryExists` | +| Insert | Write the row when none exists; **a tombstone counts as absent, so Insert over a tombstone returns `GoodEntryInserted`** (an explicit client insert may re-populate a deleted time; only the live writer's redelivery is barred, §2.2) | `GoodEntryInserted` / `BadEntryExists` | | Replace | Overwrite the row when one exists | `GoodEntryReplaced` / `BadNoEntryExists` | | Update | Write the row either way | `GoodEntryReplaced` or `GoodEntryInserted` | | DeleteAtTime | Write a tombstone (`deleted = true`, value NULL) where a row exists | `Good` / `BadNoEntryExists` | @@ -212,8 +226,12 @@ Further rules: target`. Stop early, with a Warning, if `suspended` is true. Stop at `VisibilityTimeout`. - Measured: median 0 ms, p95 6 ms, max 13 ms. Up to 7/20 inserts were not visible to an immediate re-read, so the wait is required. A concurrent writer can only raise `target`. The key-poll fallback (S11) is not used. -- Existence checks for sparse item times (Insert/Replace/Update/DeleteAtTime) use `SELECT ts, deleted FROM {p}sample - WHERE tag = $1 AND ts IN ($2, …)`, chunked at 1,000 times (T0 Q11). +- **(as built)** Existence checks for item times (Insert/Replace/Update/DeleteAtTime) are **not** `IN` lists: they use the T0 Q11 range + form, one query per `QuestDbTimeClusters` cluster, `SELECT ts FROM {p}sample WHERE tag = $1 AND ts >= $2 AND ts <= $3 AND NOT + deleted`, and the set test is client-side. The `IN ($2, …)` form this section originally proposed is not used. +- **Unknown tag (as built):** Insert and Update register the tag in `QuestDbTagCatalog`; Replace and the deletes do not (Replace ⇒ + `BadNoEntryExists`, DeleteAtTime ⇒ `BadNoEntryExists`, DeleteRaw ⇒ `BadNoData`). +- **Tombstone shape (as built):** `kind = Empty (0)`, value columns NULL, `status = BadNoData`, `deleted = true`. - **DeleteRaw (T0-confirmed), page size 1,000:** 1. `SELECT ts FROM {p}sample WHERE tag = $1 AND ts >= $2 AND ts < $3 AND NOT deleted ORDER BY ts LIMIT 1000`. 2. Write one 1,000-row tombstone INSERT. @@ -221,12 +239,17 @@ Further rules: 4. Continue from `last ts + 10 ticks`, until a page comes back empty. The wait between pages is required, or the next page re-reads rows just tombstoned. Measured: 2,500 rows, 3 pages, 103 ms. -- **Ruling candidate (T0 Q13):** an Insert, Replace or Update item timestamped before 1970-01-01 returns - `BadOutOfRange`, because QuestDB cannot store it. Deletes there find nothing and return `BadNoEntryExists`. Pre-1970 - conformance seeds are skipped for QuestDB. -- The existence checks for Insert and Replace read committed data. If a live recorder sample at the *same* - timestamp is still in flight in the WAL, it can race the updater. Updates normally target past timestamps, so this - is accepted and documented. +- **Ruling RC1 (T0 Q13):** an Insert, Replace or Update item timestamped before 1970-01-01 returns + `BadOutOfRange`, because QuestDB cannot store it (nothing is sent; a null or `MinValue` timestamp is still + `BadInvalidTimestamp`, and **as built** a pre-1970 item that is also NaN reports `BadOutOfRange`, not `BadInvalidArgument`, + because the pre-1970 filter runs first). DeleteAtTime there returns `BadNoEntryExists`; DeleteRaw finds nothing. A DeleteRaw + *window* may start before 1970. Pre-1970 conformance seeds are skipped for QuestDB. +- **Concurrency (as built).** The updater serializes calls **per tag, in-process only** (64 striped async locks, held from the + existence read through the visibility wait), so two concurrent Insert/Replace/Delete batches on one tag cannot both act on a stale + read. It does **not** lock against the live writer or a second node. The existence checks read committed data, so a live recorder + sample at the *same* timestamp still in flight in the WAL can race the updater, and **two nodes writing the same new key + concurrently can produce last-write-wins** (QuestDB has no `UPDLOCK`; the Primary gate on the recorder makes it rare). Updates + normally target past timestamps, so this residual is accepted and documented. ### 2.4 Retention @@ -242,10 +265,13 @@ T0-confirmed (Q5): formatted with `InvariantCulture`. - TTL is not allowed on the `PARTITION BY YEAR` `hist_tag` table (`TTL value must be an integer multiple of the partition size`). +- **TTL is enforced only on write commits (as built, N3).** QuestDB looks for expired partitions when a write commits to the table, + so a quiet table never drops anything; old partitions linger until the next write. - **Reading the current TTL:** `SELECT ttlValue, ttlUnit FROM tables() WHERE table_name = $1`. QuestDB **normalises the unit**: 7 days reads `1 WEEK`, 14 days `2 WEEK`, 30 days `30 DAY`, none `0 HOUR`. The bootstrapper compares in - hours (`HOUR`=1, `DAY`=24, `WEEK`=168; any other unit counts as different) and issues an `ALTER` only on a - difference. + hours (`HOUR`=1, `DAY`=24, `WEEK`=168; any other unit counts as different, RC3) and issues an `ALTER` only on a + difference. A large `RetentionDays` that QuestDB normalises to MONTH/YEAR therefore re-`ALTER`s on every boot (harmless). + `AutoCreateSchema=false` touches no TTL. - `ALTER … SET TTL` is applied through the WAL, so `tables()` shows the old value for 25–100 ms. The bootstrapper does not re-read to verify. - Expiry is checked against the **wall clock**: the image default is `cairo.ttl.use.wall.clock=true`. A future-dated @@ -261,7 +287,11 @@ T0-confirmed (Q5): Every query filters `NOT deleted`. Every value goes through `StoredSample`, so the quality clamp is the same as SQL Server's. -- **Raw.** A range on `ts`, `ORDER BY ts [DESC]`, `LIMIT @top` (`maxValuesPerNode == 0` ⇒ unbounded), at most `top` samples, `ContinuationPoint: null`; the node manager pages by re-reading from the last timestamp (plan R8). The dedup key forbids same-instant ties. Bounds follow the existing data-source contract. +- **Raw.** A range on `ts`, `ORDER BY ts [DESC]`, `LIMIT @top` (`maxValuesPerNode == 0` ⇒ unbounded), at most `top` samples, `ContinuationPoint: null`; the node manager pages by re-reading from the last timestamp (plan R8). The dedup key forbids same-instant ties. Bounds follow the existing data-source contract: forward `[start, end)`, reverse `(end, start]` + descending, and **(as built)** `start == end` reads the sample exactly at that (rounded) time. **(as built)** The data source reports + `TimestampResolutionTicks = 10`, and the node manager steps off the last drained timestamp by that amount (forward `T + 10`, reverse + `T − 10`): a 1-tick step is truncated back to `T`, so a one-value-per-page forward read would loop forever. The rule is *not* "trim + everything ≤ the last emitted", because the same paging serves the Gateway backend, where same-timestamp ties are legal. - **T0-confirmed form (review fix):** `SELECT ts, kind, num_value, int_value, str_value, status FROM {p}sample WHERE tag = $1 AND ts >= $2 AND ts < $3 AND NOT deleted ORDER BY ts [DESC] LIMIT $4`. - `$4` is a **bound** parameter, `NpgsqlDbType.Integer` (Int32) or `Bigint` (Int64); both work on 10.0.1, ASC and @@ -283,15 +313,19 @@ Server's. ``` `k` and `dir` are generated integer literals. Measured: 102 times in about 16–20 ms over 100k rows per tag, with no mismatches. `ASOF JOIN` was faster (1 ms), but it yields only the previous neighbour, so it was not chosen. - `AtTimeEvaluator` then decides the result. -- **Events.** Filtered by source and `ts` range, ordered ascending (reversed windows are swapped, as built today), - and limited to `maxEvents`. + `AtTimeEvaluator` then decides the result. **(as built, M3)** Each time is evaluated at its *rounded* value but the result is stamped + with the time the client *requested* (UTC-normalised, not truncated), as on SQL Server, so answers match requests. Stored data is µs-only, + so a sub-µs request adjacent to a sample is an exact hit. +- **Events.** Filtered by source and `ts` range `[start, end)`, ordered `(ts, event_id)` ascending (reversed windows are swapped, as built + today). **(as built)** With a bound, `maxEvents + 1` rows are fetched; the extra row is dropped and becomes the continuation point. - **Processed.** **One query per node** (T0-confirmed for every family; the S11 two-statement fallback is not needed). - Bucket index (T0-confirmed): `(cast(ts AS LONG) * 10 - $1) / $2`. - `cast(ts AS LONG)` is µs since 1970. - `$1` = the rounded window start in **ticks since the Unix epoch** (`start.Ticks − DateTime.UnixEpoch.Ticks`). - `$2` = the interval in ticks. Both are `NpgsqlDbType.Bigint`. - The result is exact because both sides are µs-aligned (R6), and it cannot overflow before year 9999. + - **(as built)** The bucket timestamp returned for a Processed bucket is the *rounded* window start plus `i × interval`, so with a + sub-µs request start it differs from SQL Server's label by the sub-µs remainder; values and membership are identical. - The query returns the full `BucketRow` statistics. Usable, good, bad and uncertain counts come from `sum(CASE …)` over the status column. - It does **not** use `SAMPLE BY`, whose alignment rules differ from ours. Computing the index ourselves keeps the @@ -345,6 +379,11 @@ Server's. The Gateway keys (`Endpoint`, `ApiKey`, `UseTls`) are not required for QuestDb, the same as for SqlServer. +**(as built) Pool size.** QuestDB's default limit is **64** PG-wire connections; Npgsql's default `Maximum Pool Size` is **100**, and the +read, write, update and alarm paths share one pool per node. Operators should set `Maximum Pool Size` in the connection string below the +server limit, per node (a redundant pair shares one QuestDB). The validator checks the prefix only when a consumer is enabled, but the +connection helper also refuses a malformed prefix (`ArgumentException`) when the first seam is built. + ## 5. Testing - **T1 refactor gate.** The SQL Server integration suite (118) and unit suite pass unchanged before and after the @@ -364,7 +403,11 @@ The Gateway keys (`Endpoint`, `ApiKey`, `UseTls`) are not required for QuestDb, - concurrent bootstrap from two instances; - intra-batch microsecond collisions; - the `hist_tag` upsert; - - transient-failure behaviour, writing against a stopped port. + - transient-failure behaviour, writing against a stopped port; + - **(as built)** pre-1970 handling (writer drop, `BadOutOfRange`, read bounds), the `long.MinValue` read repair, Insert over a tombstone, + per-tag updater serialization, and a forward cap-1 paging regression against a real QuestDB. + - Conformance differences are named overrides: pre-1970 seeds are skipped on QuestDB, and AtTime timestamp assertions compare `Round(x)` to + the requested time (identity on SQL Server). The QuestDB fixture prunes idle pool connections after 1 s because of the 64-connection limit. - **Fixture.** `HISTORIAN_QUESTDB_TEST_CONNECTION`; when it is absent, the tests skip. Each test class uses a unique throwaway `TablePrefix` (`t__`) and drops its tables on dispose. - **Host.** Registry and validator tests for `Backend=QuestDb`, beside the SqlServer ones. From 8d82d049e955ddd2186873e8a0840bddc121a944 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Sun, 11 Oct 2026 03:35:35 -0400 Subject: [PATCH 22/23] fix(historian): QuestDB final-review fix wave (I-1, M-1, M-3, M-5, M-8, M-9) - I-1: QuestDbConnection caps Npgsql Maximum Pool Size at 24 (DefaultMaxPoolSize) when the connection string does not set it; an explicit value is kept. The effective value is logged once at Information. - M-1: QuestDbTagCatalog.EnsureAsync waits for the tag table's sequenced commits and re-checks before inserting, so it no longer overwrites a provisioner row (unit/description) that is committed but not yet applied. Integration test reproduces the clobber against real QuestDB (falsified). - M-3: the pre-1970 drop Warning is rate-limited per tag (once per 10 min); DroppedPre1970 still counts every sample. - M-5: QuestDbHistorianProvider is IAsyncDisposable/IDisposable and disposes the shared Npgsql pool; Program.cs registers the provider through a factory and every seam resolves it from the container, so DI disposes it at shutdown. - M-8, M-9: docs fixes (HistoryUpdate transaction wording; folded duplicate HistoryUpdate paragraph). Pool-size docs updated for the new default cap. Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe --- docs/Configuration.md | 12 ++-- docs/Historian.md | 13 ++-- .../QuestDbConnection.cs | 44 +++++++++++- .../QuestDbHistorianProvider.cs | 30 +++++++- .../QuestDbHistorianValueWriter.cs | 30 +++++++- .../QuestDbTagCatalog.cs | 29 ++++++-- src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs | 16 +++-- .../ProvisioningTests.cs | 33 +++++++++ .../QuestDbConnectionTests.cs | 44 ++++++++++++ .../QuestDbHistorianProviderTests.cs | 17 +++++ .../QuestDbHistorianValueWriterTests.cs | 71 +++++++++++++++++++ 11 files changed, 309 insertions(+), 30 deletions(-) create mode 100644 tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests/QuestDbHistorianValueWriterTests.cs diff --git a/docs/Configuration.md b/docs/Configuration.md index 597fa8cc..4b23cf3c 100644 --- a/docs/Configuration.md +++ b/docs/Configuration.md @@ -419,19 +419,17 @@ when `ServerHistorian:Enabled` **or** `AlarmHistorian:Enabled` is true (`QuestDb | Key | Default | Validation / meaning | |---|---|---| -| `ServerHistorian:QuestDb:ConnectionString` | `""` | **Required.** Npgsql connection string for QuestDB's PG-wire port, e.g. `Host=questdb;Port=8812;Username=admin;Password=quest;Database=qdb;ServerCompatibilityMode=NoTypeLoading`. Env `ServerHistorian__QuestDb__ConnectionString` / `${secret:}` — never commit a real one. Set Npgsql `Maximum Pool Size` below QuestDB's 64-connection default limit (Npgsql's own default is 100), per node. | +| `ServerHistorian:QuestDb:ConnectionString` | `""` | **Required.** Npgsql connection string for QuestDB's PG-wire port, e.g. `Host=questdb;Port=8812;Username=admin;Password=quest;Database=qdb;ServerCompatibilityMode=NoTypeLoading`. Env `ServerHistorian__QuestDb__ConnectionString` / `${secret:}` — never commit a real one. When the string does not set Npgsql `Maximum Pool Size`, the backend caps it at **24** (Npgsql's own default of 100 exceeds QuestDB's 64-connection default limit); an explicit value is kept as written, and the effective value is logged at Information. Keep the per-node total of a pair under QuestDB's limit. | | `ServerHistorian:QuestDb:TablePrefix` | `hist_` | Must match `^[A-Za-z0-9_]+$` (spliced into DDL/DML). Tables are `{prefix}sample`, `{prefix}event`, `{prefix}tag`. | | `ServerHistorian:QuestDb:RetentionDays` | unset | `>= 1` when set. Applied as table TTL to the sample and event tables (enforced on write commits only — a quiet table never drops). Unset keeps data forever. | | `ServerHistorian:QuestDb:CommandTimeout` | `00:00:30` | Must be `> 0`; rounded up to whole seconds. | | `ServerHistorian:QuestDb:AutoCreateSchema` | `true` | `true` creates the tables and reconciles the TTL on first use; `false` only verifies the tables exist and fails every historian call (health `LastError`, no crash) if one is missing. | | `ServerHistorian:QuestDb:VisibilityTimeout` | `00:00:05` | Must be `>= 0`. How long a write waits for its rows to become visible to queries (WAL apply); `0` disables the wait. A timeout only logs a Warning. | -HistoryUpdate (soft delete) needs `ServerHistorian:Enabled=true` and a `HistoryUpdate` grant, as for `SqlServer`. Semantics, the µs timestamp rules, -pre-1970 limits, TTL, and the docker-dev overlay recipe: [`docs/Historian.md`](Historian.md#questdb-backend). - -HistoryUpdate (SqlServer and QuestDb only) additionally needs `ServerHistorian:Enabled=true` and a key whose principal holds the -`HistoryUpdate` permission — see [`docs/security.md`](security.md#historyupdate-historian-writes). Full semantics, schema, aggregates and the -docker-dev recipe: [`docs/Historian.md`](Historian.md#sql-server-backend). +HistoryUpdate (SqlServer and QuestDb only; QuestDb deletes are soft — tombstones) additionally needs `ServerHistorian:Enabled=true` and a key +whose principal holds the `HistoryUpdate` permission — see [`docs/security.md`](security.md#historyupdate-historian-writes). Full semantics, +schema, aggregates and the docker-dev recipe: [`docs/Historian.md`](Historian.md#sql-server-backend) (SqlServer) and +[`docs/Historian.md`](Historian.md#questdb-backend) (QuestDb: the µs timestamp rules, pre-1970 limits, TTL and the docker-dev overlay). #### `ContinuousHistorization` (recorder + outbox knobs) diff --git a/docs/Historian.md b/docs/Historian.md index e18e8ae1..38d9bd37 100644 --- a/docs/Historian.md +++ b/docs/Historian.md @@ -840,9 +840,11 @@ Read only when `Backend=QuestDb`; bound by `HistorianBackendRegistry.BindQuestDb | `QuestDb:VisibilityTimeout` | `00:00:05` | `>= 0` | How long a write waits for its rows to become visible to queries ([WAL visibility](#wal-visibility-and-the-read-your-writes-wait)). `0` disables the wait. | ⚠️ **Connection pool size.** QuestDB's default limit is **64** concurrent PG-wire connections (`pg.net.connection.limit`), while Npgsql's -default `Maximum Pool Size` is **100**. A busy node (HistoryRead batches, the recorder, the updater and alarm writes all share one pool) -can therefore be refused by QuestDB before Npgsql's own pool is full. Set `Maximum Pool Size` in the connection string below the server -limit — and below it **per node**, since a redundant pair shares one QuestDB: for example `…;Maximum Pool Size=24` on each of two nodes. +own default `Maximum Pool Size` is **100**. A busy node (HistoryRead batches, the recorder, the updater and alarm writes all share one pool) +could therefore be refused by QuestDB before Npgsql's pool is full. **When the connection string does not set `Maximum Pool Size` +(or `MaxPoolSize`), the backend now caps it at 24**, so the two nodes of a redundant pair sharing one QuestDB stay under the server limit +together (2 × 24 = 48). An explicit value in the connection string is always kept as written — including one above 64, since QuestDB's +limit is server-configurable. The effective value is logged once at Information when the backend first builds its pool. ### Tables @@ -876,7 +878,7 @@ through `BindTimestamp`. Consequences: timestamp by that amount (forward `T + resolution`, reverse `T − resolution`). Stepping by 1 tick would be truncated back to `T` and a one-value-per-page forward read would loop forever. Same-timestamp ties (legal on the Gateway backend) are still paged within the timestamp. - **Pre-1970 timestamps cannot be stored** (QuestDB: `designated timestamp before 1970-01-01 is not allowed`; the whole INSERT fails). So: - - the **live writer** drops such samples with one Warning per batch (counted in `DroppedPre1970`) — a retry could never succeed, and sending + - the **live writer** drops such samples with at most one Warning per tag per 10 minutes (a device with an unset clock would otherwise warn every drain pass; every dropped sample is still counted in `DroppedPre1970`) — a retry could never succeed, and sending them would push the tag into the #538 quarantine; - **HistoryUpdate** Insert / Replace / Update of such an item returns `BadOutOfRange` (nothing is sent); `DeleteAtTime` returns `BadNoEntryExists`; `DeleteRaw` finds nothing; @@ -1008,7 +1010,8 @@ batch limiter and `HistoryReadDeadline`. | `DeleteRawModifiedDetails` (`IsDeleteModified=false`) | `HistoryDeleteRawModified` | one node-level code: `Good`; `BadNoData` if `[start, end)` held nothing | | `DeleteAtTimeDetails` | `HistoryDeleteAtTime` | per time: `Good`; `BadNoEntryExists` | -Each backend call is one transaction (`MERGE … OUTPUT $action` against the TVP). The node-level `StatusCode` stays `Good` when item results are +Each backend call is applied as a unit (SqlServer: one transaction, `MERGE … OUTPUT $action` against the TVP; QuestDb: an INSERT per +chunk plus the WAL-visibility wait, no transaction — see [QuestDB backend](#questdb-backend)). The node-level `StatusCode` stays `Good` when item results are mixed (Part 11 reports item outcomes in `OperationResults`); it is Bad only for whole-node failures. **Per-item rejections** (answered without touching SQL; other items still proceed): a null item ⇒ `BadInvalidArgument`; diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbConnection.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbConnection.cs index 1b003425..5d965006 100644 --- a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbConnection.cs +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbConnection.cs @@ -1,4 +1,6 @@ using System.Net.Sockets; +using Microsoft.Extensions.Logging; +using Microsoft.Extensions.Logging.Abstractions; using Npgsql; using NpgsqlTypes; @@ -12,8 +14,15 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb; /// failures with / . Every QuestDB server error carries /// SQLSTATE 00000, so classification is by message text, never by SQLSTATE. /// -internal sealed class QuestDbConnection : IAsyncDisposable +internal sealed class QuestDbConnection : IAsyncDisposable, IDisposable { + /// + /// Npgsql pool cap applied when the connection string does not set Maximum Pool Size itself. Npgsql's own + /// default (100) exceeds QuestDB's default PG-wire connection limit (64), and the server's default read + /// concurrency alone can reach 64; an explicit value in the connection string is always kept. + /// + internal const int DefaultMaxPoolSize = 24; + private static readonly long MinSafeTicks = 10; // DateTime.MinValue itself binds as -infinity private static readonly long MaxSafeTicks = DateTime.MaxValue.Ticks - DateTime.MaxValue.Ticks % 10; // MaxValue itself binds as +infinity @@ -21,14 +30,40 @@ internal sealed class QuestDbConnection : IAsyncDisposable /// Creates the connection helper; options must already be valid. /// Validated options. - public QuestDbConnection(QuestDbHistorianOptions options) + /// Optional logger; the effective pool size is logged once at Information. + public QuestDbConnection(QuestDbHistorianOptions options, ILogger? log = null) { ArgumentNullException.ThrowIfNull(options); if (!QuestDbHistorianOptions.IsValidPrefix(options.TablePrefix)) throw new ArgumentException("Invalid table prefix.", nameof(options)); TablePrefix = options.TablePrefix; CommandTimeoutSeconds = Math.Max(1, (int)Math.Ceiling(options.CommandTimeout.TotalSeconds)); - _dataSource = new NpgsqlDataSourceBuilder(options.ConnectionString).Build(); + var csb = new NpgsqlConnectionStringBuilder(options.ConnectionString); + var explicitPool = HasExplicitMaxPoolSize(options.ConnectionString); + if (!explicitPool) csb.MaxPoolSize = DefaultMaxPoolSize; + EffectiveMaxPoolSize = csb.MaxPoolSize; + (log ?? NullLogger.Instance).LogInformation( + "QuestDB historian connection pool: Maximum Pool Size = {MaxPoolSize} ({Source}).", + EffectiveMaxPoolSize, explicitPool ? "from the connection string" : "default cap"); + _dataSource = new NpgsqlDataSourceBuilder(csb.ConnectionString).Build(); + } + + /// The Npgsql Maximum Pool Size in effect (the connection string's own value, else ). + internal int EffectiveMaxPoolSize { get; } + + // Npgsql accepts "Maximum Pool Size" and "MaxPoolSize" (case-insensitive); a generic parse keeps the caller's own + // keywords so "was it set" is answered by the string itself, not by the builder's default. + private static bool HasExplicitMaxPoolSize(string connectionString) + { + var generic = new System.Data.Common.DbConnectionStringBuilder { ConnectionString = connectionString }; + foreach (string key in generic.Keys) + { + var k = key.Replace(" ", "", StringComparison.Ordinal); + if (string.Equals(k, "MaxPoolSize", StringComparison.OrdinalIgnoreCase) + || string.Equals(k, "MaximumPoolSize", StringComparison.OrdinalIgnoreCase)) + return true; + } + return false; } /// The validated table prefix. @@ -70,6 +105,9 @@ internal sealed class QuestDbConnection : IAsyncDisposable /// public ValueTask DisposeAsync() => _dataSource.DisposeAsync(); + /// + public void Dispose() => _dataSource.Dispose(); + // ---- timestamps (R6) ------------------------------------------------------------------------------------------- /// diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianProvider.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianProvider.cs index 39af8413..f56d2205 100644 --- a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianProvider.cs +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianProvider.cs @@ -18,8 +18,10 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb; /// validation and no I/O, so resolving the provider for a backend nobody consumes never fails on its options /// (they are validated, consumer-gated, by the host's ServerHistorianOptionsValidator). The first /// Create* call's service provider supplies the logger factory the shared pieces log through. +/// The provider owns the shared connection pool and disposes it (IAsyncDisposable/IDisposable); the host +/// registers the provider through a DI factory so the container disposes it at shutdown. /// -public sealed class QuestDbHistorianProvider : IHistorianBackendProvider +public sealed class QuestDbHistorianProvider : IHistorianBackendProvider, IAsyncDisposable, IDisposable { private static readonly FrozenSet Kinds = Enum.GetValues().ToFrozenSet(); @@ -27,6 +29,7 @@ public sealed class QuestDbHistorianProvider : IHistorianBackendProvider private readonly QuestDbHistorianOptions _options; private readonly object _initLock = new(); private Shared? _shared; + private bool _disposed; /// Creates the provider over the ServerHistorian:QuestDb options (validated elsewhere). /// The ServerHistorian:QuestDb options. @@ -51,7 +54,8 @@ public sealed class QuestDbHistorianProvider : IHistorianBackendProvider { if (_shared is { } raced) return raced; var loggers = Loggers(sp); - var conn = new QuestDbConnection(_options); + ObjectDisposedException.ThrowIf(_disposed, this); + var conn = new QuestDbConnection(_options, loggers.CreateLogger()); var bootstrap = new QuestDbSchemaBootstrapper(conn, _options, loggers.CreateLogger()); var wal = new QuestDbWalVisibility(conn, _options.VisibilityTimeout, loggers.CreateLogger()); var shared = new Shared(conn, bootstrap, new QuestDbTagCatalog(conn, bootstrap, wal), wal); @@ -101,6 +105,28 @@ public sealed class QuestDbHistorianProvider : IHistorianBackendProvider Loggers(sp).CreateLogger()); } + /// Disposes the shared connection pool, when one was built. Idempotent. + /// A task completing when the pool is disposed. + public async ValueTask DisposeAsync() + { + if (TakeForDispose() is { } conn) await conn.DisposeAsync().ConfigureAwait(false); + } + + /// Disposes the shared connection pool, when one was built. Idempotent. + public void Dispose() => TakeForDispose()?.Dispose(); + + private QuestDbConnection? TakeForDispose() + { + lock (_initLock) + { + if (_disposed) return null; + _disposed = true; + var conn = _shared?.Conn; + _shared = null; + return conn; + } + } + private sealed record Shared( QuestDbConnection Conn, QuestDbSchemaBootstrapper Bootstrap, QuestDbTagCatalog Catalog, QuestDbWalVisibility Wal); } diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianValueWriter.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianValueWriter.cs index a7fba836..c9eb37d0 100644 --- a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianValueWriter.cs +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianValueWriter.cs @@ -1,3 +1,4 @@ +using System.Collections.Concurrent; using System.Text; using Microsoft.Extensions.Logging; using Microsoft.Extensions.Logging.Abstractions; @@ -19,7 +20,7 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb; /// Batch normalisation is at µs resolution (non-finite doubles become Empty /// + Bad, an in-batch µs collision keeps the first). RC1: a sample whose time is before 1970 cannot be a /// QuestDB designated timestamp and one such row would fail the whole multi-row INSERT, so those samples are dropped -/// (Warning once per batch, counted in ) and the call still succeeds for the rest. +/// (Warning at most once per tag per ; every one counted in ) and the call still succeeds for the rest. /// R5: after inserting, the writer waits for WAL visibility so an immediate redelivery sees the rows. /// Inserts are chunked at rows (8 bound parameters per row, far under the 32,767 limit). /// @@ -28,12 +29,19 @@ internal sealed class QuestDbHistorianValueWriter : IHistorianValueWriter /// Rows per multi-row INSERT (T0 Q2: 1,000 rows is reliable). internal const int BatchRows = 1000; + /// + /// Minimum time between two pre-1970 drop Warnings for the same tag. A device with an unset clock sends such + /// times on every drain pass; still counts every dropped sample. + /// + internal static readonly TimeSpan Pre1970WarningInterval = TimeSpan.FromMinutes(10); + private readonly QuestDbConnection _conn; private readonly QuestDbTagCatalog _catalog; private readonly QuestDbWalVisibility _wal; private readonly ILogger _log; private readonly Func _utcNow; private long _droppedPre1970; + private readonly ConcurrentDictionary _pre1970WarnedUtc = new(StringComparer.Ordinal); /// Creates the writer. /// Connection helper. @@ -73,7 +81,9 @@ internal sealed class QuestDbHistorianValueWriter : IHistorianValueWriter if (dropped > 0) { Interlocked.Add(ref _droppedPre1970, dropped); - _log.LogWarning("QuestDB historian dropped {Count} sample(s) for a tag because their time is before 1970-01-01 (not storable).", dropped); + if (ShouldWarnPre1970(tag)) + _log.LogWarning("QuestDB historian dropped {Count} sample(s) for a tag because their time is before 1970-01-01 (not storable); " + + "further drops for this tag are not logged for {Minutes} minutes.", dropped, Pre1970WarningInterval.TotalMinutes); } if (rows.Count == 0) return true; @@ -98,6 +108,22 @@ internal sealed class QuestDbHistorianValueWriter : IHistorianValueWriter } } + // Once per tag per Pre1970WarningInterval, measured on the writer's clock. + private bool ShouldWarnPre1970(string tag) + { + var now = _utcNow(); + var warn = false; + _pre1970WarnedUtc.AddOrUpdate(tag, + _ => { warn = true; return now; }, + (_, last) => + { + if (now - last < Pre1970WarningInterval) { warn = false; return last; } + warn = true; + return now; + }); + return warn; + } + // T0 Q11 range form, one query per cluster: tombstones are NOT filtered out (a deleted row must not be resurrected). private async Task> ExistingTicksAsync(string tag, List rows, CancellationToken ct) { diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbTagCatalog.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbTagCatalog.cs index 341ba89d..8076eded 100644 --- a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbTagCatalog.cs +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbTagCatalog.cs @@ -42,15 +42,24 @@ internal sealed class QuestDbTagCatalog if (!HistorianTagNames.IsValid(name)) return false; if (_known.ContainsKey(name)) return true; await _bootstrap.EnsureAsync(ct).ConfigureAwait(false); - await using var conn = await _conn.OpenAsync(ct).ConfigureAwait(false); - await using var cmd = _conn.CreateCommand(conn, $"SELECT count() FROM {_conn.TagTable} WHERE name = $1"); - QuestDbConnection.BindText(cmd, name); - var found = Convert.ToInt64(await cmd.ExecuteScalarAsync(ct).ConfigureAwait(false)) > 0; + var found = await StoredAsync(name, ct).ConfigureAwait(false); if (found) _known[name] = 0; return found; } - /// Registers the tag when absent (no-op when present) and waits for the row to become visible. + private async Task StoredAsync(string name, CancellationToken ct) + { + await using var conn = await _conn.OpenAsync(ct).ConfigureAwait(false); + await using var cmd = _conn.CreateCommand(conn, $"SELECT count() FROM {_conn.TagTable} WHERE name = $1"); + QuestDbConnection.BindText(cmd, name); + return Convert.ToInt64(await cmd.ExecuteScalarAsync(ct).ConfigureAwait(false)) > 0; + } + + /// + /// Registers the tag when absent (no-op when present) and waits for the row to become visible. On a miss it first + /// waits for the tag table's already-sequenced commits and re-checks, so a row the provisioner has committed but + /// QuestDB has not yet applied is found rather than overwritten (its unit/description kept). + /// /// Historian tag name. /// Initial data_type (only written on create; provisioning corrects it later). /// Cancellation. @@ -62,6 +71,16 @@ internal sealed class QuestDbTagCatalog throw new ArgumentException($"Historian tag name exceeds {HistorianTagNames.MaxLength} characters.", nameof(name)); if (await ExistsAsync(name, ct).ConfigureAwait(false)) return; + // The miss may only mean another writer's row (the provisioner's, carrying eng_unit/description) is committed + // but not yet applied. Inserting now would land AFTER it and the dedup key would replace it with a row that has + // no metadata, so first wait for everything already sequenced on the tag table and look again. + await _wal.WaitAsync(_conn.TagTable, ct).ConfigureAwait(false); + if (await StoredAsync(name, ct).ConfigureAwait(false)) + { + _known[name] = 0; + return; + } + await using (var conn = await _conn.OpenAsync(ct).ConfigureAwait(false)) { await using var cmd = _conn.CreateCommand(conn, diff --git a/src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs b/src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs index 6fa61706..416052f2 100644 --- a/src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs +++ b/src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs @@ -235,15 +235,19 @@ if (hasDriver) // Backend selection: resolve the provider once (unknown name => OptionsValidationException naming the // allowed values) and build every historian seam through it. var historianProvider = HistorianBackendRegistry.Resolve(serverHistorianOptions, builder.Configuration); - builder.Services.AddSingleton(historianProvider); + // Registered through a factory (not as an instance) so the container owns it and disposes it at shutdown + // (the QuestDb provider's shared Npgsql pool); every seam below resolves the provider from the container + // rather than capturing the local, so whichever seam builds the backend's shared state also makes the + // container track the provider for disposal. + builder.Services.AddSingleton(_ => historianProvider); // HistoryUpdate is gated like HistoryRead: None unconditionally, the real holder only when enabled. builder.Services.AddSingleton(HistorianUpdaterHolder.None); if (serverHistorianOptions.Enabled) - builder.Services.AddSingleton(sp => new HistorianUpdaterHolder(historianProvider.CreateUpdater(sp))); + builder.Services.AddSingleton(sp => new HistorianUpdaterHolder(sp.GetRequiredService().CreateUpdater(sp))); builder.Services.AddAlarmHistorian( builder.Configuration, - (_, sp) => historianProvider.CreateAlarmWriter(sp)); + (_, sp) => sp.GetRequiredService().CreateAlarmWriter(sp)); // Node-local LocalDb: caches the deployed-configuration artifact so this node can boot from // its last-known-good config when central SQL is unreachable, and optionally replicates that @@ -269,7 +273,7 @@ if (hasDriver) // node manager's HistoryRead overrides block-bridge to. Runtime owns the gating. builder.Services.AddServerHistorian( builder.Configuration, - (_, sp) => historianProvider.CreateDataSource(sp)); + (_, sp) => sp.GetRequiredService().CreateDataSource(sp)); // Config-gated historian tag provisioning. When the ServerHistorian section is enabled this registers the // selected backend's provisioner (GatewayTagProvisioner -> the gateway's EnsureTags, or the SqlServer Tag-row @@ -280,7 +284,7 @@ if (hasDriver) // the SAME ServerHistorian:Enabled flag as the read path — both use the one resolved backend. builder.Services.AddHistorianProvisioning( builder.Configuration, - (_, sp) => historianProvider.CreateProvisioning(sp)); + (_, sp) => sp.GetRequiredService().CreateProvisioning(sp)); // Continuous historization of driver (non-Galaxy) tag values. Gated on ContinuousHistorization:Enabled // AND ServerHistorian:Enabled: the recorder drains driver-tag live values through the selected backend's @@ -319,7 +323,7 @@ if (hasDriver) // The backend's live-value writer. On Gateway it opens its OWN gRPC channel to the same gateway (a second // channel to a co-located sidecar is cheap — the gateway pools the historian sessions server-side); on // SqlServer it shares the provider's connection helper, schema bootstrapper and tag catalog. - builder.Services.AddSingleton(sp => historianProvider.CreateValueWriter(sp)); + builder.Services.AddSingleton(sp => sp.GetRequiredService().CreateValueWriter(sp)); } // Bind every cross-platform driver factory before AddAkka resolves IDriverFactory — replaces diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/ProvisioningTests.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/ProvisioningTests.cs index e1b8605e..68ce7c68 100644 --- a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/ProvisioningTests.cs +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/ProvisioningTests.cs @@ -92,6 +92,39 @@ public sealed class ProvisioningTests(QuestDbFixture fx) (await TagAsync(c, "k")).ShouldBe(((byte)DriverDataType.Float64, "kg", "mass")); } + /// + /// Final review M-1: the catalog's first sighting of a tag can run while the provisioner's row is committed but + /// not yet applied (its WAL-visibility window). An unconditional catalog INSERT then lands after it and the + /// dedup key replaces the provisioner's row with one carrying no unit/description. The provisioner here does + /// not wait (VisibilityTimeout 0) and has its own catalog, so every catalog Ensure runs inside that window. + /// + [Fact] + public async Task A_catalog_Ensure_inside_the_provisioners_visibility_window_keeps_its_metadata() + { + fx.SkipIfUnavailable(); + var o = fx.Options(fx.UniquePrefix()); + await using var c = new QuestDbConnection(o); + var b = new QuestDbSchemaBootstrapper(c, o); + await b.EnsureAsync(CancellationToken.None); + var noWait = new QuestDbWalVisibility(c, TimeSpan.Zero); + var provisioner = new QuestDbTagProvisioner(c, b, new QuestDbTagCatalog(c, b, noWait), noWait); + var catalog = new QuestDbTagCatalog(c, b, new QuestDbWalVisibility(c, o.VisibilityTimeout)); + + const int Tags = 25; + for (var i = 0; i < Tags; i++) + { + var name = $"race{i}"; + (await provisioner.EnsureTagsAsync([new(name, DriverDataType.Float64, "kg", $"mass {i}")], CancellationToken.None)) + .Ensured.ShouldBe(1); + await catalog.EnsureAsync(name, 1, CancellationToken.None); // the recorder's first value for the tag + } + + await new QuestDbWalVisibility(c, o.VisibilityTimeout).WaitAsync(c.TagTable, CancellationToken.None); + for (var i = 0; i < Tags; i++) + (await TagAsync(c, $"race{i}")).ShouldBe(((byte)DriverDataType.Float64, "kg", $"mass {i}"), $"race{i}"); + (await CountAsync(c)).ShouldBe(Tags); + } + [Fact] public async Task A_connection_failure_is_a_Failed_tally_not_an_exception() { diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests/QuestDbConnectionTests.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests/QuestDbConnectionTests.cs index 0f39bf6e..afc09155 100644 --- a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests/QuestDbConnectionTests.cs +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests/QuestDbConnectionTests.cs @@ -137,4 +137,48 @@ public sealed class QuestDbConnectionTests c.EventTable.ShouldBe("p_event"); c.TagTable.ShouldBe("p_tag"); } + + // ---- pool size (final review I-1) ------------------------------------------------------------------------------- + + private static int PoolOf(string cs) + { + using var c = new QuestDbConnection(new QuestDbHistorianOptions { ConnectionString = cs }); + return c.EffectiveMaxPoolSize; + } + + [Fact] + public void An_unset_Maximum_Pool_Size_is_capped_at_24() => + PoolOf("Host=127.0.0.1;Port=1").ShouldBe(24); + + [Fact] + public void The_default_cap_constant_is_24() => QuestDbConnection.DefaultMaxPoolSize.ShouldBe(24); + + [Theory] + [InlineData("Host=127.0.0.1;Port=1;Maximum Pool Size=10", 10)] + [InlineData("Host=127.0.0.1;Port=1;Maximum Pool Size=80", 80)] + [InlineData("Host=127.0.0.1;Port=1;maximum pool size=30", 30)] + [InlineData("Host=127.0.0.1;Port=1;MaxPoolSize=80", 80)] + public void An_explicit_Maximum_Pool_Size_is_preserved(string cs, int expected) => + PoolOf(cs).ShouldBe(expected); + + [Fact] + public void The_effective_pool_size_is_logged_once_at_Information() + { + var log = new InfoLogger(); + using var c = new QuestDbConnection(new QuestDbHistorianOptions { ConnectionString = "Host=127.0.0.1;Port=1" }, log); + log.Messages.Count.ShouldBe(1); + log.Messages[0].ShouldContain("24"); + } + + private sealed class InfoLogger : Microsoft.Extensions.Logging.ILogger + { + public List Messages { get; } = []; + public IDisposable? BeginScope(TState state) where TState : notnull => null; + public bool IsEnabled(Microsoft.Extensions.Logging.LogLevel logLevel) => true; + public void Log(Microsoft.Extensions.Logging.LogLevel logLevel, Microsoft.Extensions.Logging.EventId eventId, TState state, + Exception? exception, Func formatter) + { + if (logLevel == Microsoft.Extensions.Logging.LogLevel.Information) Messages.Add(formatter(state, exception)); + } + } } diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests/QuestDbHistorianProviderTests.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests/QuestDbHistorianProviderTests.cs index b5df992b..8f3b4d5d 100644 --- a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests/QuestDbHistorianProviderTests.cs +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests/QuestDbHistorianProviderTests.cs @@ -76,6 +76,23 @@ public sealed class QuestDbHistorianProviderTests Field(w, "_wal").ShouldBeSameAs(Field(alarm, "_wal")); } + [Fact] + public async Task Disposing_the_provider_disposes_the_shared_pool_and_is_idempotent() + { + var sp = new NullSp(); + var p = Make(); + var conn = (QuestDbConnection)Field(p.CreateDataSource(sp), "_conn")!; + var pool = (Npgsql.NpgsqlDataSource)Field(conn, "_dataSource")!; + await p.DisposeAsync(); + Should.Throw(() => pool.CreateConnection().Open()); + await p.DisposeAsync(); + p.Dispose(); + Should.Throw(() => p.CreateDataSource(sp)); + } + + [Fact] + public async Task Disposing_an_unused_provider_is_a_no_op() => await Make().DisposeAsync(); + [Fact] public void The_constructor_rejects_null_options() => Should.Throw(() => new QuestDbHistorianProvider(null!)); diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests/QuestDbHistorianValueWriterTests.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests/QuestDbHistorianValueWriterTests.cs new file mode 100644 index 00000000..6650188e --- /dev/null +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests/QuestDbHistorianValueWriterTests.cs @@ -0,0 +1,71 @@ +using Microsoft.Extensions.Logging; +using Shouldly; +using Xunit; +using ZB.MOM.WW.OtOpcUa.Core.Abstractions; + +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests; + +/// +/// The pre-1970 drop Warning is rate-limited per tag (final review M-3) while the counter still counts every dropped +/// sample. All-pre-1970 batches return before any QuestDB call, so the connection points at a closed port. +/// +public sealed class QuestDbHistorianValueWriterTests +{ + private static readonly DateTime FileTimeZero = new(1601, 1, 1, 0, 0, 0, DateTimeKind.Utc); + + private static (QuestDbHistorianValueWriter W, ListLogger Log, Action Advance) Make() + { + var opts = new QuestDbHistorianOptions { ConnectionString = "Host=127.0.0.1;Port=1;Timeout=1" }; + var conn = new QuestDbConnection(opts); + var boot = new QuestDbSchemaBootstrapper(conn, opts); + var wal = new QuestDbWalVisibility(conn, opts.VisibilityTimeout); + var now = new DateTime(2026, 10, 11, 12, 0, 0, DateTimeKind.Utc); + var log = new ListLogger(); + var w = new QuestDbHistorianValueWriter(conn, new QuestDbTagCatalog(conn, boot, wal), wal, log, () => now); + return (w, log, d => now += d); + } + + private static HistorizationValue[] Pre1970(int n) => + Enumerable.Range(0, n).Select(i => new HistorizationValue(FileTimeZero.AddSeconds(i), HistorianValue.FromDouble(i), 0)).ToArray(); + + [Fact] + public async Task The_pre_1970_warning_is_logged_once_per_tag_per_interval_but_every_drop_is_counted() + { + var (w, log, advance) = Make(); + + (await w.WriteLiveValuesAsync("a", Pre1970(2), CancellationToken.None)).ShouldBeTrue(); + advance(TimeSpan.FromSeconds(2)); + (await w.WriteLiveValuesAsync("a", Pre1970(3), CancellationToken.None)).ShouldBeTrue(); + advance(QuestDbHistorianValueWriter.Pre1970WarningInterval - TimeSpan.FromSeconds(3)); + (await w.WriteLiveValuesAsync("a", Pre1970(1), CancellationToken.None)).ShouldBeTrue(); + + log.Warnings.ShouldBe(1); + w.DroppedPre1970.ShouldBe(6); + + // A different tag has its own budget. + (await w.WriteLiveValuesAsync("b", Pre1970(1), CancellationToken.None)).ShouldBeTrue(); + log.Warnings.ShouldBe(2); + + // Once the interval has passed since the last Warning for "a", it warns again. + advance(TimeSpan.FromSeconds(1)); + (await w.WriteLiveValuesAsync("a", Pre1970(1), CancellationToken.None)).ShouldBeTrue(); + log.Warnings.ShouldBe(3); + w.DroppedPre1970.ShouldBe(8); + } + + [Fact] + public void The_interval_is_ten_minutes() => + QuestDbHistorianValueWriter.Pre1970WarningInterval.ShouldBe(TimeSpan.FromMinutes(10)); + + private sealed class ListLogger : ILogger + { + private int _warnings; + public int Warnings => Volatile.Read(ref _warnings); + public IDisposable? BeginScope(TState state) where TState : notnull => null; + public bool IsEnabled(LogLevel logLevel) => true; + public void Log(LogLevel logLevel, EventId eventId, TState state, Exception? exception, Func formatter) + { + if (logLevel == LogLevel.Warning) Interlocked.Increment(ref _warnings); + } + } +} From 1569a59ad26e3a80c6a7ffdd2fd87616fba2c2b8 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Sun, 11 Oct 2026 04:00:03 -0400 Subject: [PATCH 23/23] test(historian): QuestDB live gate PASSED on docker-dev (6/6 legs); RetentionDays compose key Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe --- docker-dev/docker-compose.yml | 4 + .../2026-10-10-questdb-historian-live-gate.md | 205 ++++++++++++++++++ 2 files changed, 209 insertions(+) create mode 100644 docs/plans/2026-10-10-questdb-historian-live-gate.md diff --git a/docker-dev/docker-compose.yml b/docker-dev/docker-compose.yml index d6e439b8..42baee8d 100644 --- a/docker-dev/docker-compose.yml +++ b/docker-dev/docker-compose.yml @@ -444,6 +444,8 @@ services: # QuestDb = the QuestDB historian; the `questdb` service comes from the docker-dev/docker-compose.questdb.yml overlay # (add `-f docker-dev/docker-compose.questdb.yml` and OTOPCUA_HISTORIAN_BACKEND=QuestDb). ServerHistorian__QuestDb__ConnectionString: "${OTOPCUA_HISTORIAN_QUESTDB:-Host=questdb;Port=8812;Username=admin;Password=quest;Database=qdb;ServerCompatibilityMode=NoTypeLoading}" + # Empty = no TTL (data kept forever); a positive day count sets the sample/event table TTL. + ServerHistorian__QuestDb__RetentionDays: "${OTOPCUA_HISTORIAN_QUESTDB_RETENTION_DAYS:-}" # Continuous historization of driver tag values (Gitea #491's live gate). Gated on BOTH # ContinuousHistorization:Enabled AND ServerHistorian:Enabled — with either off no recorder is # spawned, so this stays inert on the default rig. OutboxPath is the FasterLog crash boundary and @@ -580,6 +582,8 @@ services: # QuestDb = the QuestDB historian; the `questdb` service comes from the docker-dev/docker-compose.questdb.yml overlay # (add `-f docker-dev/docker-compose.questdb.yml` and OTOPCUA_HISTORIAN_BACKEND=QuestDb). ServerHistorian__QuestDb__ConnectionString: "${OTOPCUA_HISTORIAN_QUESTDB:-Host=questdb;Port=8812;Username=admin;Password=quest;Database=qdb;ServerCompatibilityMode=NoTypeLoading}" + # Empty = no TTL (data kept forever); a positive day count sets the sample/event table TTL. + ServerHistorian__QuestDb__RetentionDays: "${OTOPCUA_HISTORIAN_QUESTDB_RETENTION_DAYS:-}" # Continuous historization — see central-1 for the full note + the gate recipe. Both centrals # carry it so the pair is symmetric; :9200 round-robins them, and the recorder must exist on # whichever node holds the drivers. diff --git a/docs/plans/2026-10-10-questdb-historian-live-gate.md b/docs/plans/2026-10-10-questdb-historian-live-gate.md new file mode 100644 index 00000000..a39d401a --- /dev/null +++ b/docs/plans/2026-10-10-questdb-historian-live-gate.md @@ -0,0 +1,205 @@ +# QuestDB historian — live gate (T9), 2026-10-11 + +**Verdict: PASSED.** All six legs of spec §7 passed on the docker-dev rig. The rig ran `feat/questdb-historian` at +`2ebe0a42`, with the central pair on image `0701f6a833ea`, rebuilt for this gate. The QuestDB backend pointed at the +overlay `questdb` service (`questdb/questdb:10.0.1`, `docker-dev/docker-compose.questdb.yml`), with default table +prefix `hist_`. + +The gate also produced two findings, neither in the QuestDB backend. Both are described under "Gate findings" below: +- A crash failover (SIGKILL of the recording Primary) leaves a recording gap of about 30 s. This is the #537 + Primary-gated capture waiting for Akka downing; it is backend-independent. +- One `libgssapi_krb5.so.2` load error is printed per central boot. It is benign. + +## Setup + +- `docker-dev/docker-compose.yml` gained `ServerHistorian__QuestDb__RetentionDays` + (`OTOPCUA_HISTORIAN_QUESTDB_RETENTION_DAYS`, default empty = no TTL) on both centrals, beside the existing + `ServerHistorian__QuestDb__ConnectionString` (`OTOPCUA_HISTORIAN_QUESTDB`). +- Central-1 and central-2 were started with: + + ``` + OTOPCUA_HISTORIAN_ENABLED=true OTOPCUA_HISTORIAN_BACKEND=QuestDb OTOPCUA_CONTHIST_ENABLED=true \ + docker compose -f docker-dev/docker-compose.yml -f docker-dev/docker-compose.questdb.yml up -d --no-deps central-1 central-2 + ``` + + With `RetentionDays` empty, both nodes booted cleanly; the empty value binds as "no TTL". Each logged + `QuestDB historian connection pool: Maximum Pool Size = 24 (default cap).` (the final-review I-1 fix). +- Cluster `MAIN` fixtures on `10.100.0.35`: opc-plc (`opcplc/plc/FastUInt1`, UInt32, about 1 Hz) and pymodbus + `standard` (`pymodbus/plc/HR5`). +- Gate configuration, written to the config DB and deployed headlessly (`deploy.sh`, deployment `0b4f5208`): + - New Calculation String tag `Calc/eng/FastLabel` = `"v" + FastUInt1` (script `MAIN-script-fastlabel`). + - `FastUInt1`, `FastLabel` and `HR5` were marked `isHistorized`. + - A UNS reference `a/l/e1/FastUInt1` was added. +- Tags were provisioned at deploy (`dispatched=3, requested=3, ensured=3, skipped=0, failed=0` on both nodes). `hist_tag`: + + | name | data_type | + |---|---| + | `Calc/eng/FastLabel` | 9 (String) | + | `opcplc/plc/FastUInt1` | 5 (Int64) | + | `pymodbus/plc/HR5` | 4 | + +- Tables at first boot, from `tables()`: + - `hist_sample`, `hist_event`: WAL, `PARTITION BY DAY`, `ttlValue 0`. + - `hist_tag`: WAL, `PARTITION BY YEAR`. +- Two data-plane API keys were minted for the gate, each with a `Cluster`-scoped grant. The ACL flag stayed off, which + is the rig default. + - `gate-historyupdate`: ReadOnly | HistoryUpdate (4239) + - `gate-writeop`: ReadOnly | WriteOperate (159) + +Direct queries used QuestDB REST: `curl -G localhost:19000/exec --data-urlencode "query=…"`. + +## 1. Recording: Primary-only, typed, real status, one row per value + +- **Rows carry real kinds and statuses:** + - `kind=2` Int64 `int_value 278` (with `num_value 278.0`), status `0`. + - `kind=4` String `v278`, status `0`. + - At startup, `kind=0` Empty with `2150760448` (`0x80320000` BadWaitingForInitialData) and `2150694912` + (`0x80310000` BadNoCommunication). These match the SQL Server gate. +- **Primary-only:** + - Central-2 was the recording Primary, and its outbox `hlog.log.0` grew. + - Central-1 logged "this node is not the recording Primary … suppressing value capture". Its `hlog.log.0` kept its + 2026-10-09 19:13 mtime and was not written. +- **One row per value**, from 07:43:20 onwards: + - 57 FastUInt1 rows over 57 distinct timestamps; 57 FastLabel rows over 57. + - All 57 pair one-to-one: same `ts`, and `str_value = 'v' + int_value`. +- **Outbox carry-over across a backend switch.** Two rows carry source time `2026-10-09T19:13:40.129127Z` (`296` / + `v296`) with `server_ts 07:42:27`, before the deploy. They were un-drained entries left in central-2's outbox by the + 2026-10-09 SQL Server gate, and they drained into QuestDB at boot. An outbox entry outlives the backend it was + captured under, by design: the outbox is backend-neutral. + +## 2. HistoryRead through the raw and UNS NodeIds + +All reads went to central-1 (`:4840`). + +- **Raw read**, 07:44:00–07:44:10, on `ns=2;s=opcplc/plc/FastUInt1` and on `ns=3;s=a/l/e1/FastUInt1`: + - Both return the identical 10 rows, `271`…`280`, all Good. + - They equal the direct query `select ts, int_value, status from hist_sample where tag=… and not deleted` exactly, + at microsecond precision (for example `07:44:00.952990Z`). +- **Average at 10 s**, 07:44:00–07:44:30, through both NodeIds: + - Both return `275.5 / 285.5 / 295.5`, all Good. + - The direct `avg(int_value)` per 10 s bucket gives `275.5 / 285.5 / 295.5`, over 10, 10 and 10 samples. + +## 3. HistoryUpdate + +These calls went to central-1 on `ns=2;s=opcplc/plc/FastUInt1`, with `-U apikey`. Each was checked by `historyread` +and by a direct query. + +| Call | Result | Confirmed | +|---|---|---| +| insert 06:00:00 = 42 (`gate-historyupdate`) | `GoodEntryInserted` | row `42`, status 0, `deleted=false` | +| repeat insert = 43 | `BadEntryExists` | row still `42` | +| replace 06:00:00 = 99 | `GoodEntryReplaced` | row `99` | +| replace 06:00:30 (no entry) | `BadNoEntryExists` | — | +| update 06:01:00 = 11 (absent) | `GoodEntryInserted` | row `11` | +| delete-at-time 06:00:00 | `Good` | tombstone row: `int_value NULL`, status `0x809B0000`, `deleted=true` | +| insert over the tombstone, 06:00:00 = 55 | `GoodEntryInserted` (ledger ruling: an insert over a tombstone is an insert) | read returns `55` | +| delete-at-time 06:00:00 again | `Good` | read returns only `06:01:00 = 11` | +| insert with `gate-writeop` (WriteOperate only) | Node result `BadUserAccessDenied` | no row | +| insert, anonymous | Node result `BadUserAccessDenied` | no row | +| insert at 1969-12-31 (RC1) | `BadOutOfRange` | no row | + +No QuestDB `UPDATE` or `DELETE` was issued: every change above is a new row under `DEDUP UPSERT KEYS(ts, tag)`. + +### A delete is not resurrected by an outbox redelivery + +This was proven with a real replay, not a simulation: +1. **07:47:26–07:47:35:** `docker stop` QuestDB. The Primary (central-1) kept capturing; its writes failed, so the + entries stayed pending in its outbox. +2. **07:47:35:** the outbox was snapshotted while QuestDB was down (commit point `commit.1303.0`). +3. **07:47:35:** QuestDB was restarted. The 16-value backlog (`476`…`491`) drained as one batch (`server_ts + 07:47:41.620267`), with no gap. This is a second result: QuestDB loss is ridden out by the outbox. +4. **07:50:45–46:** `delete-at-time` on two backlog rows, `07:47:28.950591` (`479`) and `07:47:31.949216` (`482`). + Both became tombstones (`deleted=true`, `0x809B0000`). +5. **07:50:46:** central-1 was stopped. Its outbox directory (on the `otopcua-localdb-central-1` volume) was replaced + with the pre-delivery snapshot. +6. **07:51:15:** central-1 was started. It re-drained the snapshot's entries, and the outbox commit point advanced + `1303 → 1328`. + +After the replay: +- Both tombstones are unchanged: same `server_ts 07:50:45.669146 / 07:50:46.074579`, still `deleted=true`. +- Every other row in the window kept its original `server_ts 07:47:41.620267`, so nothing was re-written. +- 24 rows over 12 keys (2 tags × 12): no duplicates. +- `historyread` 07:47:27–07:47:33 returns `478, 480, 481, 483`; `479` and `482` stay deleted. + +A first attempt used `ALTER TABLE hist_sample SUSPEND WAL` plus a SIGKILL of the Primary. It could not produce a +redelivery. The writer deliberately treats a suspended table as committed and acknowledges it (R5: rows committed to +the WAL become visible later, and are not re-sent). It logged "QuestDB table hist_sample is suspended (writerTxn 327 +< sequencerTxn 337); not waiting" and acked. That attempt exposed the crash-failover finding below. + +## 4. Failover + +- **07:50:46:** central-1 was stopped (step 5 above). Central-2 logged "now the recording Primary" at 07:50:46. +- **07:51:57:** `docker stop` central-2, then the Primary. Central-1 logged "this node is now the recording Primary … + resuming value capture" at 07:51:57, as central-2 began leaving. +- **07:52:13:** central-2 was restarted and rejoined as standby ("not the recording Primary"). Its outbox + `hlog.log.0` stayed frozen at 57856 bytes (mtime 07:51:58). Central-1's grew from 65536 to 84480. +- **Rate across both failovers was 1×.** Per-30 s FastUInt1 counts from 07:50:00 to 07:54:30: + 30, 29, 30, 30, 30, 30, 29, 29, 30. + - There was no gap at the boundary: `743`…`754` are contiguous across 07:51:57. + - There was no doubling: 267 rows over 267 distinct timestamps. + +## 5. Device loss: driver Bad sample and comms-loss outage marker + +This leg ran on Modbus (`HR5`). `otopcua-pymodbus-standard` was stopped at 07:54:40 and restarted at 07:56:51. + +| ts | kind | value | status | server_ts | +|---|---|---|---|---| +| 07:51:18.715101 | 2 | 5 | `0` | 07:51:18 | +| 07:54:51.960244 | 0 (Empty) | NULL | `0x80050000` BadCommunicationError (the driver's own Bad publish) | 07:56:13 | +| 07:56:48.652185 | 0 (Empty) | NULL | `0x80310000` BadNoCommunication (the #527 watchdog outage marker) | 07:56:48 | +| 07:56:58.688150 | 2 | 5 | `0` (recovery re-publish) | 07:56:58 | + +Reads over the outage: +- Raw `historyread` 07:50–07:58 shows Good, Bad, Bad, Good. +- Average at 120 s, 07:52–07:58, gives `(null) Bad` / `(null) Bad` / `5 Uncertain`. The outage is visible and is not + bridged. + +The Bad sample's `server_ts` is 82 s after its source time. Its source time is the driver's failed-read stamp; it +reached the recorder later. Its position in the series is correct either way, because ordering and reads key on +source time. + +## 6. TTL visible in table metadata + +Both centrals were recreated with `OTOPCUA_HISTORIAN_QUESTDB_RETENTION_DAYS=30`. At 07:57:41 the bootstrapper logged +`Applied TTL 30 day(s) to QuestDB table hist_sample.` and the same for `hist_event`. The TTL was applied to the +already existing tables. + +`tables()` before and after: + +| table | before | after | +|---|---|---| +| `hist_event` | 0 HOUR | 30 DAY | +| `hist_sample` | 0 HOUR | 30 DAY | +| `hist_tag` | 0 HOUR | 0 HOUR (no TTL on the tag catalog, by design) | + +`SHOW CREATE TABLE hist_sample`: +`… timestamp(ts) PARTITION BY DAY TTL 30 DAYS DEDUP UPSERT KEYS(ts,tag);` + +Recording resumed after the restart: 57 FastUInt1 rows and 56 FastLabel rows in the first minute. + +## Gate findings + +1. **A crash failover leaves a recording gap of about 30 s. This is backend-independent and predates this work.** + - **What happened:** after `docker kill` (SIGKILL) of the recording Primary at 07:45:58, the next FastUInt1 row + is at 07:46:29. Central-1 became the recording Primary only at 07:46:30, when the killed node rejoined as a new + incarnation. + - **Why:** #537 gates capture to the Primary. A crashed Primary stays unreachable but not yet downed for the Akka + failure-detector window plus `auto-down-unreachable-after = 15s`, so no node captures during that time. + - **Contrast:** a clean `docker stop` hands over at once (leg 4). The SQL Server gate only exercised the clean + path. + - **Status:** proposed follow-up issue, not fixed here. +2. **`Cannot load library libgssapi_krb5.so.2` is printed once per central boot.** It does not appear on the site + nodes, which have neither a ConfigDb nor a QuestDB client. It is most likely Npgsql's default GSS-encryption probe, + which falls back cleanly. Everything worked. If it is confirmed as Npgsql, `GSS Encryption Mode=Disable` in the + connection string silences it. + +## Restore + +- The rig was restored afterwards: + - central-1 and central-2 were recreated with the compose defaults (historian off, `Backend=Gateway`). + - The gate tag, script, UNS reference, API keys and ACL rows were removed. + - The `HR5` and `FastUInt1` TagConfigs were reverted. + - The configuration was redeployed. + - The overlay `questdb` container was stopped and removed; the `otopcua-questdb-data` volume was kept. + - The opc-plc and pymodbus fixtures were stopped (normally down). +- The compose keys stay: `Backend` defaults to `Gateway`, and `RetentionDays` defaults to empty.