diff --git a/docs/plans/2026-10-10-questdb-historian-design.md b/docs/plans/2026-10-10-questdb-historian-design.md new file mode 100644 index 00000000..c01068be --- /dev/null +++ b/docs/plans/2026-10-10-questdb-historian-design.md @@ -0,0 +1,250 @@ +# QuestDB historian backend — design + +**Status:** approved section by section on 2026-10-10. Gitea #666. +**Builds on:** `docs/plans/2026-10-09-pluggable-historian-design.md`, which defines the provider seam, typed values, the +HistoryUpdate service and the SQL Server backend, all merged to master as `8ec7a83c`. + +## 0. Decisions (user, 2026-10-10) + +| Topic | Decision | +|---|---| +| HistoryUpdate deletes | **Soft delete.** Rows carry a `deleted` flag. A delete writes a tombstone at the same `(tag, ts)`. Every read and aggregate excludes tombstones. All five HistoryUpdate operations are supported. | +| Write path | **Postgres wire protocol only, via `Npgsql`.** It is already centrally pinned at 10.0.3. No ILP, no new NuGet package. | +| Retention | **Included.** `RetentionDays` is applied as table TTL. QuestDB drops expired day partitions itself. | +| Approach | **B: shared dialect-free core plus per-backend SQL.** The pure logic moves out of the SQL Server backend into `Driver.Historian.Sql.Common`. One conformance suite runs against both backends. | +| Timestamps | **Microseconds.** Postgres-wire timestamps are microsecond resolution, and real source timestamps are at best milliseconds. Two samples less than 1 µs apart within a batch collapse to one, and the first wins. This is documented. | + +Out of scope: +- event HistoryUpdate; +- Part 11 modified values (#668); +- time-weighted aggregates (#671); +- ILP ingest; +- QuestDB authentication beyond the connection string; +- any change to the Gateway backend. + +## 1. Components + +### 1.1 `ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common` (new, no I/O) + +This is extracted from `Driver.Historian.SqlServer` with **no behaviour change**: + +- **`StoredSample`.** It holds the time, the value, the stored status and a deleted flag. It classifies a sample as + usable, good, bad or uncertain. It applies the quality-contract clamp: a NULL value carrying a non-Bad status is + served as `BadNoData`. It also defines the info-bit masks (`CodeBitsMask`, `SeverityBadBit`, `SeverityMask`). +- **`AtTimeEvaluator`.** This is the former `SqlServerHistorianDataSource.AtTimeValue`. It covers interpolation and + stepping, the outage rule (previous neighbour Bad ⇒ null plus that code; previous usable and next Bad ⇒ hold the + previous value as `UncertainDataSubNormal`), stepping forward after the last sample, `T > now` ⇒ Uncertain, and + masking `& ~0x3` before the info bits are set. +- **`BucketRow` + `AggregateEvaluator`.** This is the former `Evaluate`. Each backend's SQL produces the **same + per-bucket statistics**: total count, usable/good/bad/uncertain counts, numeric sum/min/max, first and last value + and time, the time of the min and of the max, and so on. The shared evaluator maps a `BucketRow` to the OPC UA + value and status for each of the 13 supported aggregates. Unsupported aggregates throw + `HistoryAggregateNotSupportedException`. +- **`BucketPlan`.** It handles window normalisation (a reversed window is swapped and returned descending, as built + today), interval 0 ⇒ one bucket, the 100,000-bucket cap, and tick arithmetic. It exposes the bucket start and width + in **ticks** and in **microseconds**. +- **`HistorianColumnMapping`.** It maps a `HistorianValueKind` to the stored data-type code (the former + `SqlHistorianDataTypes.FromKind`). It also maps a `HistorianValue` to its value columns (num/int/str) and back. +- **`HistoryUpdateResults`.** These are the per-item result rules for Insert, Replace, Update, DeleteAtTime and + DeleteRaw given "did a live row exist". They include the NaN/Inf rule (rejected `BadInvalidArgument` on update, + stored as `BadOutOfRange` Empty on the live write) and the rule rejecting an Empty value with a non-Bad status. +- **Shared constants.** The named status consts that both backends emit (`BadOutOfRange` and others) move into one + parity-visible `*StatusCodes` class here. Backend-specific codes stay with their backend. + +`Driver.Historian.SqlServer` then keeps only its T-SQL, the connection, the bootstrapper, the tag catalog and the +seam classes, all calling the shared core. + +### 1.2 `ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb` (new) + +| Class | Responsibility | +|---|---| +| `QuestDbHistorianOptions` | Binds `ServerHistorian:QuestDb` (§4) and validates it. | +| `QuestDbConnection` | Npgsql `NpgsqlDataSource` (pooled), command timeouts, transient-error classification, UTC timestamp reads. | +| `QuestDbSchemaBootstrapper` | Idempotent `CREATE TABLE IF NOT EXISTS` for the three tables, then applies TTL. It is concurrency-safe across two nodes starting together, and caches a failure for 5 s. | +| `QuestDbHistorianValueWriter` | `IHistorianValueWriter` for live samples (§2.2). | +| `QuestDbTagProvisioner` | `IHistorianProvisioning`: upserts `hist_tag` rows. | +| `QuestDbAlarmHistorianWriter` | `IAlarmHistorianWriter`: writes events with the deterministic SHA-256 `event_id` and the same classify rules as SQL Server (unknown ⇒ RetryPlease). | +| `QuestDbHistorianDataSource` | `IHistorianDataSource` (§3), with QuestDB SQL for raw, at-time, events and bucket statistics. | +| `QuestDbHistorianDataUpdater` | `IHistorianDataUpdater` (§2.3), the soft-delete updater. | +| `QuestDbHistorianProvider` | `IHistorianBackendProvider`: `Name = "QuestDb"`, `SupportedKinds` = all six (FrozenSet), one shared connection and bootstrapper. | + +Host changes: +- `HistorianBackendRegistry` gains `"QuestDb"`. +- `ServerHistorianOptionsValidator` validates the QuestDb section, gated on consumers, the same way it gates + SqlServer. +- `Host.csproj` gets a ProjectReference. + +## 2. Schema and writes + +### 2.1 Tables + +QuestDB has no schemas, so the tables carry a configurable `TablePrefix` (default `hist_`, `[A-Za-z0-9_]+`). All +three are **WAL tables with `DEDUP UPSERT KEYS`**, so re-sending a row with the same key replaces it. + +```sql +CREATE TABLE IF NOT EXISTS hist_sample ( + ts TIMESTAMP, tag SYMBOL, kind BYTE, num DOUBLE, int LONG, str VARCHAR, + status LONG, deleted BOOLEAN, server_ts TIMESTAMP +) TIMESTAMP(ts) PARTITION BY DAY WAL DEDUP UPSERT KEYS(ts, tag) [TTL n DAYS]; + +CREATE TABLE IF NOT EXISTS hist_event ( + ts TIMESTAMP, event_id VARCHAR, source SYMBOL, + -- plus the remaining hist.Event columns one-for-one (docs/historian/sqlserver/001_initial.sql), QuestDB-typed +) TIMESTAMP(ts) PARTITION BY DAY WAL DEDUP UPSERT KEYS(ts, event_id) [TTL n DAYS]; + +CREATE TABLE IF NOT EXISTS hist_tag ( + ts TIMESTAMP, name SYMBOL, data_type BYTE, eng_unit VARCHAR, description VARCHAR, updated_utc TIMESTAMP +) TIMESTAMP(ts) PARTITION BY YEAR WAL DEDUP UPSERT KEYS(ts, name); -- ts is always 1970-01-01, which makes this an upsert by name +``` + +Column details: +- `status` holds the uint status code as a LONG. +- `int` also holds Booleans as 0/1, matching SQL Server. +- `kind` is `HistorianValueKind`. +- The exact column types and the TTL syntax are confirmed by T0 against the pinned image. T0 amends this section + with what it finds. + +### 2.2 Live writes (recorder) — insert only if absent, the same as SQL Server + +1. Group the batch by tag and remove duplicates within the batch. **The first value wins.** A timestamp is rounded + to microseconds before this step, so microsecond collisions also collapse. +2. For each tag, fetch the timestamps that already exist in the batch's `[min ts, max ts]` range. **Tombstones are + included**, so a re-sent outbox batch can never resurrect a deleted row or overwrite a HistoryUpdate change. +3. Insert only the new rows, in one parameterised multi-row `INSERT` per tag group. +4. Stored-value rules are shared with SQL Server: + - NaN/±Inf, or any Empty with a non-Bad status, is stored Empty with `BadOutOfRange` or `BadNoData`. + - Comms-loss outage markers are stored as given. + - A tag name over 400 characters is a per-tag failure. +5. Any SQL or transport failure makes the write return `false`, so the recorder retries and the #538 per-tag + quarantine still applies. It never throws. + +The recorder is Primary-only (#537), so step 2 does not race another live writer. The recorder does not wait for +WAL visibility. + +### 2.3 HistoryUpdate (soft delete) + +Every operation is an `INSERT`, and the dedup upsert performs the replace. QuestDB `UPDATE` and `DELETE` are never +issued. "Exists" means a row that is present **and not deleted**. + +| Operation | Effect | Result | +|---|---|---| +| Insert | Write the row when none exists; a tombstone counts as absent | `GoodEntryInserted` / `BadEntryExists` | +| Replace | Overwrite the row when one exists | `GoodEntryReplaced` / `BadNoEntryExists` | +| Update | Write the row either way | `GoodEntryReplaced` or `GoodEntryInserted` | +| DeleteAtTime | Write a tombstone (`deleted = true`, value NULL) where a row exists | `Good` / `BadNoEntryExists` | +| DeleteRaw | Write a tombstone for every live row in the range | the same codes SQL Server returns for the same cases | + +Further rules: +- The per-item rules come from `HistoryUpdateResults`, shared with SQL Server, so the codes are identical. +- Item order is preserved, with exactly one code per item. +- **Read-your-writes.** After its inserts, the updater polls `wal_tables()` until the table's applied transaction + reaches the sequencer transaction it produced, up to `VisibilityTimeout`. On a timeout the write is kept, the codes + are reported as normal, and a Warning is logged. +- The existence checks for Insert and Replace read committed data. If a live recorder sample at the *same* + timestamp is still in flight in the WAL, it can race the updater. Updates normally target past timestamps, so this + is accepted and documented. + +### 2.4 Retention + +When `RetentionDays` is set, the bootstrapper applies TTL to `hist_sample` and `hist_event`, and re-applies it when +the configured value changes. QuestDB then drops expired day partitions, tombstones included. When unset, data is +kept forever. `hist_tag` never expires. + +## 3. Reads + +Every query filters `NOT deleted`. Every value goes through `StoredSample`, so the quality clamp is the same as SQL +Server's. + +- **Raw.** A range on `ts`, `ORDER BY ts [DESC]`, `LIMIT n+1` to detect another page. The dedup key forbids + same-instant ties, so the continuation is just the last timestamp. Bounds follow the existing data-source contract. +- **AtTime.** Find the previous-or-equal neighbour and the next neighbour of each requested time. The default is + `LIMIT 1` seeks on `ts`, batched into chunked statements; T0 picks `ASOF JOIN` for the previous neighbour if it + measures better. `AtTimeEvaluator` then decides the result. +- **Events.** Filtered by source and `ts` range, ordered ascending (reversed windows are swapped, as built today), + and limited to `maxEvents`. +- **Processed.** **One query per node.** + - Bucket index: `(cast(ts AS LONG) - @startMicros) / @intervalMicros`, using microsecond values from `BucketPlan`, + grouped by the index. + - The query returns the full `BucketRow` statistics. Usable, good, bad and uncertain counts come from + `sum(CASE …)` over the status column. + - It does **not** use `SAMPLE BY`, whose alignment rules differ from ours. Computing the index ourselves keeps the + bucket edges identical to SQL Server: start inclusive, end exclusive, a partial final bucket, interval 0, and the + 100k cap. + - The time of the min or max value needs an arg-min style lookup. It is computed with a window function, or with a + second statement if QuestDB lacks one; T0 decides. `AggregateEvaluator` then produces the values. +- **Errors.** A failed read fails only that node, under the same data-source contract as SQL Server. + +## 4. Configuration (`ServerHistorian:QuestDb`) + +| Key | Default | Validation | +|---|---|---| +| `ConnectionString` | — | Required when `Backend=QuestDb` and a consumer is enabled. An Npgsql string, e.g. `Host=questdb;Port=8812;Username=admin;Password=quest;Database=qdb;ServerCompatibilityMode=NoTypeLoading` (T0 confirms the exact keywords). | +| `TablePrefix` | `hist_` | `^[A-Za-z0-9_]+$`. Only checked when a consumer is enabled. | +| `RetentionDays` | unset | ≥ 1 when set | +| `CommandTimeout` | `00:00:30` | > 0 | +| `AutoCreateSchema` | `true` | — | +| `VisibilityTimeout` | `00:00:05` | ≥ 0. 0 disables the wait. | + +The Gateway keys (`Endpoint`, `ApiKey`, `UseTls`) are not required for QuestDb, the same as for SqlServer. + +## 5. Testing + +- **T1 refactor gate.** The SQL Server integration suite (118) and unit suite pass unchanged before and after the + extraction. Each moved piece of logic is falsified once against its existing test. +- **Conformance suite** (`tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Conformance`, a class library of abstract + xUnit tests). Each backend's integration project subclasses it and supplies a fixture and an `IConformanceSeeder`, + which writes rows directly in that backend's schema. + - It covers raw forward, reverse and paging; AtTime, including the outage, after-last and future rules; all 13 + aggregates with the existing hand-computed literals; the quality contract; insert-if-absent writes and + redelivery; the HistoryUpdate result codes; and over-long tag names. + - Each legitimate backend difference is a named, documented override, never a silent skip. Microsecond rounding is + the only expected one. +- **QuestDB-only tests:** + - a tombstone hides the row and a redelivery does not resurrect it; + - the read-your-writes wait; + - TTL is applied, verified through `tables()` metadata; + - concurrent bootstrap from two instances; + - intra-batch microsecond collisions; + - the `hist_tag` upsert; + - transient-failure behaviour, writing against a stopped port. +- **Fixture.** `HISTORIAN_QUESTDB_TEST_CONNECTION`; when it is absent, the tests skip. Each test class uses a unique + throwaway `TablePrefix` (`t__`) and drops its tables on dispose. +- **Host.** Registry and validator tests for `Backend=QuestDb`, beside the SqlServer ones. + +## 6. Environment + +- docker-dev gains a `questdb` service, using an exact image tag that T0 pins (at least two weeks old). It sits behind + a compose profile (`questdb`) so the default rig is unchanged. +- It exposes PG wire `8812` and the web console `9000` on host ports that T0 chooses so they don't collide with + AdminUI `:9000`/`:9200`. +- It uses a named volume. +- The central nodes gain `ServerHistorian__QuestDb__ConnectionString` (`OTOPCUA_HISTORIAN_QUESTDB`), defaulting to the + `questdb` service. +- No new NuGet packages. + +## 7. Live gate (T9, run by the controller) + +On docker-dev, with `Backend=QuestDb`: +1. Only the Primary records, with real kinds and status, one row per value. +2. Raw and Processed HistoryRead return identical results through the raw and UNS NodeIds, and match a direct + QuestDB query. +3. HistoryUpdate covers insert, duplicate, replace, replace-missing and delete-at-time, each confirmed by a read and + by a direct query. After a delete, an outbox redelivery does not bring the row back. A WriteOperate-only key and an + anonymous session are both denied. +4. On failover, recording continues at 1×. +5. Modbus device loss stores a `BadCommunicationError` Empty sample, then the `BadNoCommunication` outage marker, + then Good on recovery. +6. TTL is visible in table metadata when `RetentionDays` is set. + +Evidence goes in `docs/plans/2026-10-10-questdb-historian-live-gate.md`, and the rig is restored afterwards. + +## 8. Work order + +| Wave | Task | Model | +|---|---|---| +| W0 | **T0 spike.** Pin the image, add the compose service, and prove every QuestDB assumption here: Npgsql parameterised multi-row inserts, `DEDUP UPSERT`, `wal_tables()` visibility, TTL syntax, microsecond timestamps over Npgsql, the min/max-time query form, AtTime neighbour lookup, concurrent `CREATE TABLE IF NOT EXISTS`. **Amend §2–§4 with the findings before W1.** | controller + sonnet | +| W1 | T1: extract `Sql.Common`, leaving SQL Server's behaviour unchanged | opus | +| W2 | T2 conformance suite over SQL Server ‖ T3 QuestDB skeleton (options, connection, bootstrapper, fixture) | sonnet ‖ sonnet | +| W3 | T4 writes, provisioning and alarms ‖ T5 reads and aggregates ‖ T6 updater with soft delete | sonnet ‖ opus ‖ sonnet | +| W4 | T7 provider, registry and validator, plus the conformance suite run against QuestDB | sonnet | +| W5 | T8 docs (Historian.md, Configuration.md, CLAUDE.md, docker-dev README), then a Fable whole-branch review, one fix wave, and T9 live gate | sonnet / fable / opus / controller |