docs(historian): QuestDB historian backend design (#666)

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe

docs/plans/2026-10-10-questdb-historian-design.md

@@ -0,0 +1,250 @@
+# QuestDB historian backend — design
+
+**Status:** approved section by section on 2026-10-10. Gitea #666.
+**Builds on:** `docs/plans/2026-10-09-pluggable-historian-design.md`, which defines the provider seam, typed values, the
+HistoryUpdate service and the SQL Server backend, all merged to master as `8ec7a83c`.
+
+## 0. Decisions (user, 2026-10-10)
+
+| Topic | Decision |
+|---|---|
+| HistoryUpdate deletes | **Soft delete.** Rows carry a `deleted` flag. A delete writes a tombstone at the same `(tag, ts)`. Every read and aggregate excludes tombstones. All five HistoryUpdate operations are supported. |
+| Write path | **Postgres wire protocol only, via `Npgsql`.** It is already centrally pinned at 10.0.3. No ILP, no new NuGet package. |
+| Retention | **Included.** `RetentionDays` is applied as table TTL. QuestDB drops expired day partitions itself. |
+| Approach | **B: shared dialect-free core plus per-backend SQL.** The pure logic moves out of the SQL Server backend into `Driver.Historian.Sql.Common`. One conformance suite runs against both backends. |
+| Timestamps | **Microseconds.** Postgres-wire timestamps are microsecond resolution, and real source timestamps are at best milliseconds. Two samples less than 1 µs apart within a batch collapse to one, and the first wins. This is documented. |
+
+Out of scope:
+- event HistoryUpdate;
+- Part 11 modified values (#668);
+- time-weighted aggregates (#671);
+- ILP ingest;
+- QuestDB authentication beyond the connection string;
+- any change to the Gateway backend.
+
+## 1. Components
+
+### 1.1 `ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common` (new, no I/O)
+
+This is extracted from `Driver.Historian.SqlServer` with **no behaviour change**:
+
+- **`StoredSample`.** It holds the time, the value, the stored status and a deleted flag. It classifies a sample as
+  usable, good, bad or uncertain. It applies the quality-contract clamp: a NULL value carrying a non-Bad status is
+  served as `BadNoData`. It also defines the info-bit masks (`CodeBitsMask`, `SeverityBadBit`, `SeverityMask`).
+- **`AtTimeEvaluator`.** This is the former `SqlServerHistorianDataSource.AtTimeValue`. It covers interpolation and
+  stepping, the outage rule (previous neighbour Bad ⇒ null plus that code; previous usable and next Bad ⇒ hold the
+  previous value as `UncertainDataSubNormal`), stepping forward after the last sample, `T > now` ⇒ Uncertain, and
+  masking `& ~0x3` before the info bits are set.
+- **`BucketRow` + `AggregateEvaluator`.** This is the former `Evaluate`. Each backend's SQL produces the **same
+  per-bucket statistics**: total count, usable/good/bad/uncertain counts, numeric sum/min/max, first and last value
+  and time, the time of the min and of the max, and so on. The shared evaluator maps a `BucketRow` to the OPC UA
+  value and status for each of the 13 supported aggregates. Unsupported aggregates throw
+  `HistoryAggregateNotSupportedException`.
+- **`BucketPlan`.** It handles window normalisation (a reversed window is swapped and returned descending, as built
+  today), interval 0 ⇒ one bucket, the 100,000-bucket cap, and tick arithmetic. It exposes the bucket start and width
+  in **ticks** and in **microseconds**.
+- **`HistorianColumnMapping`.** It maps a `HistorianValueKind` to the stored data-type code (the former
+  `SqlHistorianDataTypes.FromKind`). It also maps a `HistorianValue` to its value columns (num/int/str) and back.
+- **`HistoryUpdateResults`.** These are the per-item result rules for Insert, Replace, Update, DeleteAtTime and
+  DeleteRaw given "did a live row exist". They include the NaN/Inf rule (rejected `BadInvalidArgument` on update,
+  stored as `BadOutOfRange` Empty on the live write) and the rule rejecting an Empty value with a non-Bad status.
+- **Shared constants.** The named status consts that both backends emit (`BadOutOfRange` and others) move into one
+  parity-visible `*StatusCodes` class here. Backend-specific codes stay with their backend.
+
+`Driver.Historian.SqlServer` then keeps only its T-SQL, the connection, the bootstrapper, the tag catalog and the
+seam classes, all calling the shared core.
+
+### 1.2 `ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb` (new)
+
+| Class | Responsibility |
+|---|---|
+| `QuestDbHistorianOptions` | Binds `ServerHistorian:QuestDb` (§4) and validates it. |
+| `QuestDbConnection` | Npgsql `NpgsqlDataSource` (pooled), command timeouts, transient-error classification, UTC timestamp reads. |
+| `QuestDbSchemaBootstrapper` | Idempotent `CREATE TABLE IF NOT EXISTS` for the three tables, then applies TTL. It is concurrency-safe across two nodes starting together, and caches a failure for 5 s. |
+| `QuestDbHistorianValueWriter` | `IHistorianValueWriter` for live samples (§2.2). |
+| `QuestDbTagProvisioner` | `IHistorianProvisioning`: upserts `hist_tag` rows. |
+| `QuestDbAlarmHistorianWriter` | `IAlarmHistorianWriter`: writes events with the deterministic SHA-256 `event_id` and the same classify rules as SQL Server (unknown ⇒ RetryPlease). |
+| `QuestDbHistorianDataSource` | `IHistorianDataSource` (§3), with QuestDB SQL for raw, at-time, events and bucket statistics. |
+| `QuestDbHistorianDataUpdater` | `IHistorianDataUpdater` (§2.3), the soft-delete updater. |
+| `QuestDbHistorianProvider` | `IHistorianBackendProvider`: `Name = "QuestDb"`, `SupportedKinds` = all six (FrozenSet), one shared connection and bootstrapper. |
+
+Host changes:
+- `HistorianBackendRegistry` gains `"QuestDb"`.
+- `ServerHistorianOptionsValidator` validates the QuestDb section, gated on consumers, the same way it gates
+  SqlServer.
+- `Host.csproj` gets a ProjectReference.
+
+## 2. Schema and writes
+
+### 2.1 Tables
+
+QuestDB has no schemas, so the tables carry a configurable `TablePrefix` (default `hist_`, `[A-Za-z0-9_]+`). All
+three are **WAL tables with `DEDUP UPSERT KEYS`**, so re-sending a row with the same key replaces it.
+
+```sql
+CREATE TABLE IF NOT EXISTS hist_sample (
+  ts TIMESTAMP, tag SYMBOL, kind BYTE, num DOUBLE, int LONG, str VARCHAR,
+  status LONG, deleted BOOLEAN, server_ts TIMESTAMP
+) TIMESTAMP(ts) PARTITION BY DAY WAL DEDUP UPSERT KEYS(ts, tag) [TTL n DAYS];
+
+CREATE TABLE IF NOT EXISTS hist_event (
+  ts TIMESTAMP, event_id VARCHAR, source SYMBOL,
+  -- plus the remaining hist.Event columns one-for-one (docs/historian/sqlserver/001_initial.sql), QuestDB-typed
+) TIMESTAMP(ts) PARTITION BY DAY WAL DEDUP UPSERT KEYS(ts, event_id) [TTL n DAYS];
+
+CREATE TABLE IF NOT EXISTS hist_tag (
+  ts TIMESTAMP, name SYMBOL, data_type BYTE, eng_unit VARCHAR, description VARCHAR, updated_utc TIMESTAMP
+) TIMESTAMP(ts) PARTITION BY YEAR WAL DEDUP UPSERT KEYS(ts, name);  -- ts is always 1970-01-01, which makes this an upsert by name
+```
+
+Column details:
+- `status` holds the uint status code as a LONG.
+- `int` also holds Booleans as 0/1, matching SQL Server.
+- `kind` is `HistorianValueKind`.
+- The exact column types and the TTL syntax are confirmed by T0 against the pinned image. T0 amends this section
+  with what it finds.
+
+### 2.2 Live writes (recorder) — insert only if absent, the same as SQL Server
+
+1. Group the batch by tag and remove duplicates within the batch. **The first value wins.** A timestamp is rounded
+   to microseconds before this step, so microsecond collisions also collapse.
+2. For each tag, fetch the timestamps that already exist in the batch's `[min ts, max ts]` range. **Tombstones are
+   included**, so a re-sent outbox batch can never resurrect a deleted row or overwrite a HistoryUpdate change.
+3. Insert only the new rows, in one parameterised multi-row `INSERT` per tag group.
+4. Stored-value rules are shared with SQL Server:
+   - NaN/±Inf, or any Empty with a non-Bad status, is stored Empty with `BadOutOfRange` or `BadNoData`.
+   - Comms-loss outage markers are stored as given.
+   - A tag name over 400 characters is a per-tag failure.
+5. Any SQL or transport failure makes the write return `false`, so the recorder retries and the #538 per-tag
+   quarantine still applies. It never throws.
+
+The recorder is Primary-only (#537), so step 2 does not race another live writer. The recorder does not wait for
+WAL visibility.
+
+### 2.3 HistoryUpdate (soft delete)
+
+Every operation is an `INSERT`, and the dedup upsert performs the replace. QuestDB `UPDATE` and `DELETE` are never
+issued. "Exists" means a row that is present **and not deleted**.
+
+| Operation | Effect | Result |
+|---|---|---|
+| Insert | Write the row when none exists; a tombstone counts as absent | `GoodEntryInserted` / `BadEntryExists` |
+| Replace | Overwrite the row when one exists | `GoodEntryReplaced` / `BadNoEntryExists` |
+| Update | Write the row either way | `GoodEntryReplaced` or `GoodEntryInserted` |
+| DeleteAtTime | Write a tombstone (`deleted = true`, value NULL) where a row exists | `Good` / `BadNoEntryExists` |
+| DeleteRaw | Write a tombstone for every live row in the range | the same codes SQL Server returns for the same cases |
+
+Further rules:
+- The per-item rules come from `HistoryUpdateResults`, shared with SQL Server, so the codes are identical.
+- Item order is preserved, with exactly one code per item.
+- **Read-your-writes.** After its inserts, the updater polls `wal_tables()` until the table's applied transaction
+  reaches the sequencer transaction it produced, up to `VisibilityTimeout`. On a timeout the write is kept, the codes
+  are reported as normal, and a Warning is logged.
+- The existence checks for Insert and Replace read committed data. If a live recorder sample at the *same*
+  timestamp is still in flight in the WAL, it can race the updater. Updates normally target past timestamps, so this
+  is accepted and documented.
+
+### 2.4 Retention
+
+When `RetentionDays` is set, the bootstrapper applies TTL to `hist_sample` and `hist_event`, and re-applies it when
+the configured value changes. QuestDB then drops expired day partitions, tombstones included. When unset, data is
+kept forever. `hist_tag` never expires.
+
+## 3. Reads
+
+Every query filters `NOT deleted`. Every value goes through `StoredSample`, so the quality clamp is the same as SQL
+Server's.
+
+- **Raw.** A range on `ts`, `ORDER BY ts [DESC]`, `LIMIT n+1` to detect another page. The dedup key forbids
+  same-instant ties, so the continuation is just the last timestamp. Bounds follow the existing data-source contract.
+- **AtTime.** Find the previous-or-equal neighbour and the next neighbour of each requested time. The default is
+  `LIMIT 1` seeks on `ts`, batched into chunked statements; T0 picks `ASOF JOIN` for the previous neighbour if it
+  measures better. `AtTimeEvaluator` then decides the result.
+- **Events.** Filtered by source and `ts` range, ordered ascending (reversed windows are swapped, as built today),
+  and limited to `maxEvents`.
+- **Processed.** **One query per node.**
+  - Bucket index: `(cast(ts AS LONG) - @startMicros) / @intervalMicros`, using microsecond values from `BucketPlan`,
+    grouped by the index.
+  - The query returns the full `BucketRow` statistics. Usable, good, bad and uncertain counts come from
+    `sum(CASE …)` over the status column.
+  - It does **not** use `SAMPLE BY`, whose alignment rules differ from ours. Computing the index ourselves keeps the
+    bucket edges identical to SQL Server: start inclusive, end exclusive, a partial final bucket, interval 0, and the
+    100k cap.
+  - The time of the min or max value needs an arg-min style lookup. It is computed with a window function, or with a
+    second statement if QuestDB lacks one; T0 decides. `AggregateEvaluator` then produces the values.
+- **Errors.** A failed read fails only that node, under the same data-source contract as SQL Server.
+
+## 4. Configuration (`ServerHistorian:QuestDb`)
+
+| Key | Default | Validation |
+|---|---|---|
+| `ConnectionString` | — | Required when `Backend=QuestDb` and a consumer is enabled. An Npgsql string, e.g. `Host=questdb;Port=8812;Username=admin;Password=quest;Database=qdb;ServerCompatibilityMode=NoTypeLoading` (T0 confirms the exact keywords). |
+| `TablePrefix` | `hist_` | `^[A-Za-z0-9_]+$`. Only checked when a consumer is enabled. |
+| `RetentionDays` | unset | ≥ 1 when set |
+| `CommandTimeout` | `00:00:30` | > 0 |
+| `AutoCreateSchema` | `true` | — |
+| `VisibilityTimeout` | `00:00:05` | ≥ 0. 0 disables the wait. |
+
+The Gateway keys (`Endpoint`, `ApiKey`, `UseTls`) are not required for QuestDb, the same as for SqlServer.
+
+## 5. Testing
+
+- **T1 refactor gate.** The SQL Server integration suite (118) and unit suite pass unchanged before and after the
+  extraction. Each moved piece of logic is falsified once against its existing test.
+- **Conformance suite** (`tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Conformance`, a class library of abstract
+  xUnit tests). Each backend's integration project subclasses it and supplies a fixture and an `IConformanceSeeder`,
+  which writes rows directly in that backend's schema.
+  - It covers raw forward, reverse and paging; AtTime, including the outage, after-last and future rules; all 13
+    aggregates with the existing hand-computed literals; the quality contract; insert-if-absent writes and
+    redelivery; the HistoryUpdate result codes; and over-long tag names.
+  - Each legitimate backend difference is a named, documented override, never a silent skip. Microsecond rounding is
+    the only expected one.
+- **QuestDB-only tests:**
+  - a tombstone hides the row and a redelivery does not resurrect it;
+  - the read-your-writes wait;
+  - TTL is applied, verified through `tables()` metadata;
+  - concurrent bootstrap from two instances;
+  - intra-batch microsecond collisions;
+  - the `hist_tag` upsert;
+  - transient-failure behaviour, writing against a stopped port.
+- **Fixture.** `HISTORIAN_QUESTDB_TEST_CONNECTION`; when it is absent, the tests skip. Each test class uses a unique
+  throwaway `TablePrefix` (`t_<guid>_`) and drops its tables on dispose.
+- **Host.** Registry and validator tests for `Backend=QuestDb`, beside the SqlServer ones.
+
+## 6. Environment
+
+- docker-dev gains a `questdb` service, using an exact image tag that T0 pins (at least two weeks old). It sits behind
+  a compose profile (`questdb`) so the default rig is unchanged.
+- It exposes PG wire `8812` and the web console `9000` on host ports that T0 chooses so they don't collide with
+  AdminUI `:9000`/`:9200`.
+- It uses a named volume.
+- The central nodes gain `ServerHistorian__QuestDb__ConnectionString` (`OTOPCUA_HISTORIAN_QUESTDB`), defaulting to the
+  `questdb` service.
+- No new NuGet packages.
+
+## 7. Live gate (T9, run by the controller)
+
+On docker-dev, with `Backend=QuestDb`:
+1. Only the Primary records, with real kinds and status, one row per value.
+2. Raw and Processed HistoryRead return identical results through the raw and UNS NodeIds, and match a direct
+   QuestDB query.
+3. HistoryUpdate covers insert, duplicate, replace, replace-missing and delete-at-time, each confirmed by a read and
+   by a direct query. After a delete, an outbox redelivery does not bring the row back. A WriteOperate-only key and an
+   anonymous session are both denied.
+4. On failover, recording continues at 1×.
+5. Modbus device loss stores a `BadCommunicationError` Empty sample, then the `BadNoCommunication` outage marker,
+   then Good on recovery.
+6. TTL is visible in table metadata when `RetentionDays` is set.
+
+Evidence goes in `docs/plans/2026-10-10-questdb-historian-live-gate.md`, and the rig is restored afterwards.
+
+## 8. Work order
+
+| Wave | Task | Model |
+|---|---|---|
+| W0 | **T0 spike.** Pin the image, add the compose service, and prove every QuestDB assumption here: Npgsql parameterised multi-row inserts, `DEDUP UPSERT`, `wal_tables()` visibility, TTL syntax, microsecond timestamps over Npgsql, the min/max-time query form, AtTime neighbour lookup, concurrent `CREATE TABLE IF NOT EXISTS`. **Amend §2–§4 with the findings before W1.** | controller + sonnet |
+| W1 | T1: extract `Sql.Common`, leaving SQL Server's behaviour unchanged | opus |
+| W2 | T2 conformance suite over SQL Server ‖ T3 QuestDB skeleton (options, connection, bootstrapper, fixture) | sonnet ‖ sonnet |
+| W3 | T4 writes, provisioning and alarms ‖ T5 reads and aggregates ‖ T6 updater with soft delete | sonnet ‖ opus ‖ sonnet |
+| W4 | T7 provider, registry and validator, plus the conformance suite run against QuestDB | sonnet |
+| W5 | T8 docs (Historian.md, Configuration.md, CLAUDE.md, docker-dev README), then a Fable whole-branch review, one fix wave, and T9 live gate | sonnet / fable / opus / controller |