lmxopcua/docs/plans/2026-10-10-questdb-historian-design.md

# QuestDB historian backend — design

**Status:** approved section by section on 2026-10-10. Gitea #666. **As-built pass (T8, 2026-10-11):** every section below was re-checked against
the merged code; where the two disagreed the text was corrected inline and marked **(as built)**. User documentation: `docs/Historian.md` §"QuestDB backend".
**Builds on:** `docs/plans/2026-10-09-pluggable-historian-design.md`, which defines the provider seam, typed values, the
HistoryUpdate service and the SQL Server backend, all merged to master as `8ec7a83c`.

## 0. Decisions (user, 2026-10-10)

| Topic | Decision |
|---|---|
| HistoryUpdate deletes | **Soft delete.** Rows carry a `deleted` flag. A delete writes a tombstone at the same `(tag, ts)`. Every read and aggregate excludes tombstones. All five HistoryUpdate operations are supported. |
| Write path | **Postgres wire protocol only, via `Npgsql`.** It is already centrally pinned at 10.0.3. No ILP, no new NuGet package. |
| Retention | **Included.** `RetentionDays` is applied as table TTL. QuestDB drops expired day partitions itself. |
| Approach | **B: shared dialect-free core plus per-backend SQL.** The pure logic moves out of the SQL Server backend into `Driver.Historian.Sql.Common`. One conformance suite runs against both backends. |
| Timestamps | **Microseconds.** Postgres-wire timestamps are microsecond resolution, and real source timestamps are at best milliseconds. Two samples less than 1 µs apart collapse to one, and the first wins — within a batch, and also against an already-stored row (the existence check runs on the rounded time). This is documented. |

Out of scope:
- event HistoryUpdate;
- Part 11 modified values (#668);
- time-weighted aggregates (#671);
- ILP ingest;
- QuestDB authentication beyond the connection string;
- any change to the Gateway backend.

## 1. Components

### 1.1 `ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common` (new, no I/O)

This is extracted from `Driver.Historian.SqlServer` with **no behaviour change**:

- **`StoredSample`.** It holds the time, the value, the stored status. It classifies a sample as
  usable, good, bad or uncertain. It applies the quality-contract clamp: a NULL value carrying a non-Bad status is
  served as `BadNoData`. It also defines the info-bit masks (`CodeBitsMask`, `SeverityBadBit`, `SeverityMask`).
- **`AtTimeEvaluator`.** This is the former `SqlServerHistorianDataSource.AtTimeValue`. It covers interpolation and
  stepping, the outage rule (previous neighbour Bad ⇒ null plus that code; previous usable and next Bad ⇒ hold the
  previous value as `UncertainDataSubNormal`), stepping forward after the last sample, `T > now` ⇒ Uncertain, and
  masking `& ~0x3` before the info bits are set.
- **`BucketRow` + `AggregateEvaluator`.** This is the former `Evaluate`. Each backend's SQL produces the **same
  per-bucket statistics**: total count, usable/good/bad/uncertain counts, numeric sum/min/max, first and last value
  and time, the time of the min and of the max, and so on. The shared evaluator maps a `BucketRow` to the OPC UA
  value and status for each of the 13 supported aggregates. Unsupported aggregates throw
  `HistoryAggregateNotSupportedException`.
- **`BucketPlan`.** It handles window normalisation (a reversed window is swapped and returned descending, as built
  today), interval 0 ⇒ one bucket, the 100,000-bucket cap, and tick arithmetic. It exposes the bucket start and width
  in **ticks** and in **microseconds**.
- **`HistorianColumnMapping`.** It maps a `HistorianValueKind` to the stored data-type code (the former
  `SqlHistorianDataTypes.FromKind`). It also maps a `HistorianValue` to its value columns (num/int/str) and back.
- **`HistoryUpdateResults`.** These are the per-item result rules for Insert, Replace, Update, DeleteAtTime and
  DeleteRaw given "did a live row exist". They include the NaN/Inf rule (rejected `BadInvalidArgument` on update,
  stored as `BadOutOfRange` Empty on the live write) and the rule rejecting an Empty value with a non-Bad status.
- **Shared constants.** The named status consts that both backends emit (`BadOutOfRange` and others) move into one
  parity-visible `*StatusCodes` class here. Backend-specific codes stay with their backend.

`Driver.Historian.SqlServer` then keeps only its T-SQL, the connection, the bootstrapper, the tag catalog and the
seam classes, all calling the shared core.

### 1.2 `ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb` (new)

| Class | Responsibility |
|---|---|
| `QuestDbHistorianOptions` | Binds `ServerHistorian:QuestDb` (§4) and validates it. |
| `QuestDbConnection` | Npgsql `NpgsqlDataSource` (pooled), command timeouts, transient-error classification, UTC timestamp reads. |
| `QuestDbSchemaBootstrapper` | Idempotent `CREATE TABLE IF NOT EXISTS` for the three tables, then applies TTL. It is concurrency-safe across two nodes starting together, and caches a failure for 5 s. |
| `QuestDbHistorianValueWriter` | `IHistorianValueWriter` for live samples (§2.2). |
| `QuestDbTagCatalog` | **(as built)** The only "is this tag known" authority (R7): a `hist_tag` row, positive results cached. Reads never create a tag; the live writer and HistoryUpdate Insert/Update do (Replace and the deletes do not). Re-inserting a known tag is skipped, because the dedup upsert would erase the unit and description the provisioner wrote. |
| `QuestDbWalVisibility` | **(as built)** The read-your-writes wait (§2.3), used by every write path. |
| `QuestDbTimeClusters` | **(as built)** Groups request times into ranges (gap > 1 min starts a new one) so existence reads are one range query per cluster. |
| `QuestDbTables`, `QuestDbAggregateQueries` | **(as built)** DDL / TTL helpers, and the push-down aggregate SQL (§3). |
| `QuestDbTagProvisioner` | `IHistorianProvisioning`: upserts `hist_tag` rows in chunks; **(as built)** a request without a unit or description keeps the stored one. |
| `QuestDbAlarmHistorianWriter` | `IAlarmHistorianWriter`: **(as built)** one INSERT per event (not batched), with the deterministic SHA-256 `event_id`. Outcomes: Ack; `RetryPlease` for transient and unclassified errors; `PermanentFail` only for a pre-1970 time or a server message naming an inconvertible / out-of-range / too-long / null value. |
| `QuestDbHistorianDataSource` | `IHistorianDataSource` (§3), with QuestDB SQL for raw, at-time, events and bucket statistics. |
| `QuestDbHistorianDataUpdater` | `IHistorianDataUpdater` (§2.3), the soft-delete updater. |
| `QuestDbHistorianProvider` | `IHistorianBackendProvider`: `Name = "QuestDb"`, `SupportedKinds` = all six (FrozenSet), one shared connection and bootstrapper. |

Host changes:
- `HistorianBackendRegistry` gains `"QuestDb"`.
- `ServerHistorianOptionsValidator` validates the QuestDb section, gated on consumers, the same way it gates
  SqlServer.
- `Host.csproj` gets a ProjectReference.
- **(as built)** `IHistorianDataSource` gains the default member `TimestampResolutionTicks` (1; QuestDB 10), and the node manager's Raw paging steps off a drained timestamp by it (§3).
- **(as built)** The provider builds its shared pieces (connection, bootstrapper, catalog, WAL wait) on the first `Create*` call, not in the constructor, so a provider nobody consumes never fails on its options.

## 2. Schema and writes

### 2.1 Tables

QuestDB has no schemas, so the tables carry a configurable `TablePrefix` (default `hist_`, `[A-Za-z0-9_]+`). All
three are **WAL tables with `DEDUP UPSERT KEYS`**, so re-sending a row with the same key replaces it.

**T0-confirmed** against `questdb/questdb:10.0.1` (`docs/plans/2026-10-10-questdb-spike.md` §DDL, Q3, Q5, Q10). The
`TTL n DAYS` clause goes **between `PARTITION BY` and `WAL`**. Placed after the dedup clause it fails with `unexpected
token [TTL]`. It is emitted only when `RetentionDays` is set, as an inlined integer, because it cannot be a parameter.

```sql
CREATE TABLE IF NOT EXISTS hist_sample (
  ts TIMESTAMP, tag SYMBOL, kind BYTE, num_value DOUBLE, int_value LONG, str_value VARCHAR,
  status LONG, deleted BOOLEAN, server_ts TIMESTAMP
) TIMESTAMP(ts) PARTITION BY DAY [TTL n DAYS] WAL DEDUP UPSERT KEYS(ts, tag);

CREATE TABLE IF NOT EXISTS hist_event (
  ts TIMESTAMP, event_id VARCHAR, source SYMBOL, alarm_id VARCHAR, alarm_name VARCHAR, alarm_type_name VARCHAR,
  severity INT, event_kind VARCHAR, message VARCHAR, user_name VARCHAR, comment VARCHAR, received_utc TIMESTAMP
) TIMESTAMP(ts) PARTITION BY DAY [TTL n DAYS] WAL DEDUP UPSERT KEYS(ts, event_id);

CREATE TABLE IF NOT EXISTS hist_tag (
  ts TIMESTAMP, name SYMBOL, data_type BYTE, eng_unit VARCHAR, description VARCHAR, updated_utc TIMESTAMP
) TIMESTAMP(ts) PARTITION BY YEAR WAL DEDUP UPSERT KEYS(ts, name);  -- ts is always 1970-01-01, which makes this an upsert by name
```

`hist_event` maps `hist.Event` one-for-one:
- `EventTimeUtc` → `ts`, `EventId` → `event_id`, `SourceName` → `source`;
- `AlarmId`/`AlarmName`/`AlarmTypeName` → `alarm_id`/`alarm_name`/`alarm_type_name`;
- `Severity` → `severity`, `EventKind` → `event_kind`, `Message` → `message`;
- `UserName` → `user_name`, `Comment` → `comment`, `ReceivedTimeUtc` → `received_utc`.

`event_id VARCHAR` works as a dedup key (T0 Q3), so re-sending an event replaces it. The S4 fallback (no dedup on
`hist_event`) is not used.

Column details (**T0-confirmed**):
- `status` holds the uint status code **unsigned** in a LONG (0..4294967295, never negative). So "severity Bad" is
  `status >= 2147483648`, not `< 0` as in SQL Server's signed `int`.
- `int_value` also holds Booleans as 0/1, matching SQL Server.
- `kind` is `HistorianValueKind` in a `BYTE`.
- Read-back CLR types:

  | Column type | Npgsql type name | CLR type / reader call |
  |---|---|---|
  | `BYTE` | `smallint` | `Int16` (`GetInt16`) |
  | `SYMBOL`, `VARCHAR` | `character varying` | `String` |
  | `LONG` | `bigint` | `Int64` |
  | `INT` | `integer` | `Int32` |
  | `BOOLEAN` | `boolean` | `Boolean` |
  | `DOUBLE` | `double precision` | `Double` |
  | `TIMESTAMP` | `timestamp without time zone` | `DateTime`, **`Kind=Unspecified`**; the reader applies `SpecifyKind(…, Utc)` |

- **NULL semantics:**
  - DOUBLE NaN and ±Inf read back as NULL.
  - `long.MinValue` (LONG) and `int.MinValue` (INT) are QuestDB's NULL sentinels.
  - `BYTE` and `BOOLEAN` have **no NULL**: unset or DBNull reads `0` / `false`. So `deleted` defaults to `false`, and a
    LEFT-JOINed "no row" yields `kind = 0`. Absence must be detected on a NULL `ts`, not on `kind`.
- **Ruling RC2 (as built):** an Int64 sample of exactly `long.MinValue` stores `int_value` as NULL. The QuestDB read
  path repairs it before the shared `FromColumns`: `kind = Int64 ∧ int_value IS NULL ∧ num_value = -2^63` ⇒
  `long.MinValue`. That double is exact. Without the repair the sample would be served as 0.
- A designated timestamp **before 1970-01-01 cannot be inserted** (`designated timestamp before 1970-01-01 is not
  allowed`). See §2.2 (RC1).
- Tag names: SYMBOL and VARCHAR take 400 characters (non-ASCII included). SYMBOL grows past its capacity hint
  (16,384) with no error, and was tested with 30,000 distinct tags. No `CAPACITY` clause is needed.

### 2.2 Live writes (recorder) — insert only if absent, the same as SQL Server

1. Group the batch by tag and remove duplicates within the batch. **The first value wins.** A timestamp is rounded
   to microseconds before this step, so microsecond collisions also collapse.
2. For each tag, fetch the timestamps that already exist. **(as built)** The batch's times are grouped into clusters
   (`QuestDbTimeClusters`: a gap over 1 minute starts a new range; the live writer also caps a range at 1,000 times), with one range query per cluster, so one
   far-away outlier does not widen a single query over every stored row in between. **Tombstones are included**, so a re-sent
   outbox batch can never resurrect a deleted row or overwrite a HistoryUpdate change. T0-confirmed form (Q11): `SELECT ts
   FROM {p}sample WHERE tag = $1 AND ts >= $2 AND ts <= $3`, with no `NOT deleted`, followed by a client-side set test. The tag is
   registered in `QuestDbTagCatalog` first (a write creates an unknown tag).
3. Insert only the new rows with parameterised multi-row `INSERT … VALUES (…),(…)`, **at most 1,000 rows per
   statement** (T0-confirmed Q2). QuestDB reads the Bind parameter count as a signed int16, so the hard limit is
   32,767 parameters. **(as built)** The live writer binds 8 parameters per row (`deleted` is the literal `false`; 4,095 rows
   would fit), the updater 9 (3,640 rows). Within one statement a duplicate key keeps the **last** row, which is
   why step 1's "first wins" is done client-side.
4. Stored-value rules are shared with SQL Server:
   - NaN/±Inf, or any Empty with a non-Bad status, is stored Empty with `BadOutOfRange` or `BadNoData`.
   - Comms-loss outage markers are stored as given.
   - A tag name over 400 characters is a per-tag failure.
   - **Ruling RC1 (T0 Q13):** a sample timestamped before 1970-01-01 cannot be stored, because QuestDB rejects
     the whole INSERT. The writer drops such samples before the INSERT, logs one Warning per batch and counts them
     (`DroppedPre1970`); the call still succeeds for the rest. A retry could
     never succeed, and sending the sample would push the tag's whole group into the #538 quarantine.
5. Any SQL or transport failure makes the write return `false`, so the recorder retries and the #538 per-tag
   quarantine still applies. It never throws.

The recorder is Primary-only (#537), so step 2 does not race another live writer. The recorder waits for WAL
visibility, bounded by `VisibilityTimeout` (plan R5), using the §2.3 wait.

Binding (T0-confirmed Q1/Q6/Q13):
- Every timestamp parameter is `NpgsqlDbType.TimestampTz` with `DateTimeKind.Utc`, after `Round` (**as built:** Unspecified is
  taken as UTC, Local is converted, then ticks are truncated to a multiple of 10). `BindTimestamp(…, clampToEpoch: true)` is used
  only for write binds of a designated timestamp; read bounds are never clamped (a pre-1970 bound is valid, and clamping it would
  move the bound across the epoch row). `Timestamp` + `Utc`
  throws in Npgsql. Npgsql truncates sub-µs ticks **toward 2000-01-01**, which rounds *up* before 2000. A raw
  `DateTime.MinValue`/`MaxValue` is sent as ±infinity, and a `[Min, Max)` range then silently matches nothing. R6's
  `BindTimestamp` (round; `MinValue` → `+10` ticks; `MaxValue` → `−9` ticks) fixes all three.
- The other parameter types:

  | Column | `NpgsqlDbType` |
  |---|---|
  | `tag`/`str_value`/other strings | `Varchar` |
  | `kind`/`data_type` | `Smallint` |
  | `num_value` | `Double` |
  | `int_value`/`status` | `Bigint` |
  | `deleted` | `Boolean` |
  | `severity` | `Integer` |

  NULL is `DBNull.Value` with the column's type.
- Every QuestDB server error carries **SQLSTATE `00000`**. `QuestDbConnection`'s classification therefore treats a
  `PostgresException` as a server-side rejection, judged by `MessageText`. Any other `NpgsqlException`, `IOException`,
  `SocketException` or timeout is transport (transient).

### 2.3 HistoryUpdate (soft delete)

Every operation is an `INSERT`, and the dedup upsert performs the replace. QuestDB `UPDATE` and `DELETE` are never
issued. "Exists" means a row that is present **and not deleted**.

| Operation | Effect | Result |
|---|---|---|
| Insert | Write the row when none exists; **a tombstone counts as absent, so Insert over a tombstone returns `GoodEntryInserted`** (an explicit client insert may re-populate a deleted time; only the live writer's redelivery is barred, §2.2) | `GoodEntryInserted` / `BadEntryExists` |
| Replace | Overwrite the row when one exists | `GoodEntryReplaced` / `BadNoEntryExists` |
| Update | Write the row either way | `GoodEntryReplaced` or `GoodEntryInserted` |
| DeleteAtTime | Write a tombstone (`deleted = true`, value NULL) where a row exists | `Good` / `BadNoEntryExists` |
| DeleteRaw | Write a tombstone for every live row in the range | the same codes SQL Server returns for the same cases |

Further rules:
- The per-item rules come from `HistoryUpdateResults`, shared with SQL Server, so the codes are identical.
- Item order is preserved, with exactly one code per item.
- **Read-your-writes.** After its inserts, the updater polls `wal_tables()` until the table's applied transaction
  reaches the sequencer transaction it produced, up to `VisibilityTimeout`. On a timeout the write is kept, the codes
  are reported as normal, and a Warning is logged.
- **The visibility wait (`QuestDbWalVisibility`, T0-confirmed Q4)** is used by the writer and the updater (R5):
  1. After the INSERT(s) return: `SELECT sequencerTxn FROM wal_tables() WHERE name = $1` (`$1` = the full table name,
     `Varchar`) → `target`. Each autocommitted INSERT is exactly one sequencer transaction.
  2. Poll `SELECT writerTxn, suspended FROM wal_tables() WHERE name = $1` about every 10 ms until `writerTxn >=
     target`. Stop early, with a Warning, if `suspended` is true. Stop at `VisibilityTimeout`.
  - Measured: median 0 ms, p95 6 ms, max 13 ms. Up to 7/20 inserts were not visible to an immediate re-read, so the
    wait is required. A concurrent writer can only raise `target`. The key-poll fallback (S11) is not used.
- **(as built)** Existence checks for item times (Insert/Replace/Update/DeleteAtTime) are **not** `IN` lists: they use the T0 Q11 range
  form, one query per `QuestDbTimeClusters` cluster, `SELECT ts FROM {p}sample WHERE tag = $1 AND ts >= $2 AND ts <= $3 AND NOT
  deleted`, and the set test is client-side. The `IN ($2, …)` form this section originally proposed is not used.
- **Unknown tag (as built):** Insert and Update register the tag in `QuestDbTagCatalog`; Replace and the deletes do not (Replace ⇒
  `BadNoEntryExists`, DeleteAtTime ⇒ `BadNoEntryExists`, DeleteRaw ⇒ `BadNoData`).
- **Tombstone shape (as built):** `kind = Empty (0)`, value columns NULL, `status = BadNoData`, `deleted = true`.
- **DeleteRaw (T0-confirmed), page size 1,000:**
  1. `SELECT ts FROM {p}sample WHERE tag = $1 AND ts >= $2 AND ts < $3 AND NOT deleted ORDER BY ts LIMIT 1000`.
  2. Write one 1,000-row tombstone INSERT.
  3. Run the visibility wait.
  4. Continue from `last ts + 10 ticks`, until a page comes back empty.
  The wait between pages is required, or the next page re-reads rows just tombstoned. Measured: 2,500 rows, 3 pages,
  103 ms.
- **Ruling RC1 (T0 Q13):** an Insert, Replace or Update item timestamped before 1970-01-01 returns
  `BadOutOfRange`, because QuestDB cannot store it (nothing is sent; a null or `MinValue` timestamp is still
  `BadInvalidTimestamp`, and **as built** a pre-1970 item that is also NaN reports `BadOutOfRange`, not `BadInvalidArgument`,
  because the pre-1970 filter runs first). DeleteAtTime there returns `BadNoEntryExists`; DeleteRaw finds nothing. A DeleteRaw
  *window* may start before 1970. Pre-1970 conformance seeds are skipped for QuestDB.
- **Concurrency (as built).** The updater serializes calls **per tag, in-process only** (64 striped async locks, held from the
  existence read through the visibility wait), so two concurrent Insert/Replace/Delete batches on one tag cannot both act on a stale
  read. It does **not** lock against the live writer or a second node. The existence checks read committed data, so a live recorder
  sample at the *same* timestamp still in flight in the WAL can race the updater, and **two nodes writing the same new key
  concurrently can produce last-write-wins** (QuestDB has no `UPDLOCK`; the Primary gate on the recorder makes it rare). Updates
  normally target past timestamps, so this residual is accepted and documented.

### 2.4 Retention

When `RetentionDays` is set, the bootstrapper applies TTL to `hist_sample` and `hist_event`, and re-applies it when
the configured value changes. QuestDB then drops expired day partitions, tombstones included. When unset, data is
kept forever. `hist_tag` never expires.

T0-confirmed (Q5):
- **Create:** `… PARTITION BY DAY TTL n DAYS WAL …` (§2.1).
- **Change:** `ALTER TABLE {p}sample SET TTL n DAYS`. **Remove** (`RetentionDays` unset while the table has a TTL):
  `ALTER TABLE … SET TTL 0 DAYS`.
- `n` cannot be a `$` parameter (`invalid syntax, should be <number> <unit> but was $1`). It is the validated integer,
  formatted with `InvariantCulture`.
- TTL is not allowed on the `PARTITION BY YEAR` `hist_tag` table (`TTL value must be an integer multiple of the
  partition size`).
- **TTL is enforced only on write commits (as built, N3).** QuestDB looks for expired partitions when a write commits to the table,
  so a quiet table never drops anything; old partitions linger until the next write.
- **Reading the current TTL:** `SELECT ttlValue, ttlUnit FROM tables() WHERE table_name = $1`. QuestDB **normalises
  the unit**: 7 days reads `1 WEEK`, 14 days `2 WEEK`, 30 days `30 DAY`, none `0 HOUR`. The bootstrapper compares in
  hours (`HOUR`=1, `DAY`=24, `WEEK`=168; any other unit counts as different, RC3) and issues an `ALTER` only on a
  difference. A large `RetentionDays` that QuestDB normalises to MONTH/YEAR therefore re-`ALTER`s on every boot (harmless).
  `AutoCreateSchema=false` touches no TTL.
- `ALTER … SET TTL` is applied through the WAL, so `tables()` shows the old value for 25–100 ms. The bootstrapper does
  not re-read to verify.
- Expiry is checked against the **wall clock**: the image default is `cairo.ttl.use.wall.clock=true`. A future-dated
  sample does not expire current data (T0 Q5c).
- **Table listing:** `SELECT table_name FROM tables() WHERE starts_with(table_name, $1)`. Never `LIKE 'hist_%'`,
  because `_` is a wildcard.
- **Concurrent bootstrap:** `CREATE TABLE IF NOT EXISTS` from 4 connections × 20 rounds produced no errors. The S11
  defence stays: catch a `PostgresException` whose `MessageText` contains `already exists`, re-check `tables()`, and
  rethrow if the table is absent. Never key on SQLSTATE (always `00000`).

## 3. Reads

Every query filters `NOT deleted`. Every value goes through `StoredSample`, so the quality clamp is the same as SQL
Server's.

- **Raw.** A range on `ts`, `ORDER BY ts [DESC]`, `LIMIT @top` (`maxValuesPerNode == 0` ⇒ unbounded), at most `top` samples, `ContinuationPoint: null`; the node manager pages by re-reading from the last timestamp (plan R8). The dedup key forbids same-instant ties. Bounds follow the existing data-source contract: forward `[start, end)`, reverse `(end, start]`
  descending, and **(as built)** `start == end` reads the sample exactly at that (rounded) time. **(as built)** The data source reports
  `TimestampResolutionTicks = 10`, and the node manager steps off the last drained timestamp by that amount (forward `T + 10`, reverse
  `T − 10`): a 1-tick step is truncated back to `T`, so a one-value-per-page forward read would loop forever. The rule is *not* "trim
  everything ≤ the last emitted", because the same paging serves the Gateway backend, where same-timestamp ties are legal.
  - **T0-confirmed form (review fix):** `SELECT ts, kind, num_value, int_value, str_value, status FROM {p}sample WHERE
    tag = $1 AND ts >= $2 AND ts < $3 AND NOT deleted ORDER BY ts [DESC] LIMIT $4`.
  - `$4` is a **bound** parameter, `NpgsqlDbType.Integer` (Int32) or `Bigint` (Int64); both work on 10.0.1, ASC and
    DESC. Use `Integer`, since `top` is an `int`.
  - `int.MaxValue` (the R8 "unbounded" value) and `long.MaxValue` both return every row. `LIMIT 0` returns no rows,
    but R8 never sends 0.
  - The inlined-literal fallback is not needed.
- **AtTime (T0-confirmed Q9).** Find the previous-or-equal neighbour and the next neighbour of each requested time
  with `LIMIT 1` seeks, `UNION ALL`-ed, **at most 50 request times (100 branches) per statement**. The exact QuestDB
  10.0.1 limit is **126 `UNION ALL` branches per statement (63 request times)**: 126 work, and 127 fail with `SQL
  model is too complex to evaluate`. So 50 times leaves a 26-branch margin. Per time `k`, with `$1` = tag and `$j` = the
  rounded time:
  ```sql
  SELECT * FROM (SELECT k AS req, 0 AS dir, ts, kind, num_value, int_value, str_value, status FROM {p}sample
    WHERE tag = $1 AND ts <= $j AND NOT deleted ORDER BY ts DESC LIMIT 1)
  UNION ALL
  SELECT * FROM (SELECT k AS req, 1 AS dir, ts, kind, num_value, int_value, str_value, status FROM {p}sample
    WHERE tag = $1 AND ts > $j AND NOT deleted ORDER BY ts ASC LIMIT 1)
  ```
  `k` and `dir` are generated integer literals. Measured: 102 times in about 16–20 ms over 100k rows per tag, with no
  mismatches. `ASOF JOIN` was faster (1 ms), but it yields only the previous neighbour, so it was not chosen.
  `AtTimeEvaluator` then decides the result. **(as built, M3)** Each time is evaluated at its *rounded* value but the result is stamped
  with the time the client *requested* (UTC-normalised, not truncated), as on SQL Server, so answers match requests. Stored data is µs-only,
  so a sub-µs request adjacent to a sample is an exact hit.
- **Events.** Filtered by source and `ts` range `[start, end)`, ordered `(ts, event_id)` ascending (reversed windows are swapped, as built
  today). **(as built)** With a bound, `maxEvents + 1` rows are fetched; the extra row is dropped and becomes the continuation point.
- **Processed.** **One query per node** (T0-confirmed for every family; the S11 two-statement fallback is not needed).
  - Bucket index (T0-confirmed): `(cast(ts AS LONG) * 10 - $1) / $2`.
    - `cast(ts AS LONG)` is µs since 1970.
    - `$1` = the rounded window start in **ticks since the Unix epoch** (`start.Ticks − DateTime.UnixEpoch.Ticks`).
    - `$2` = the interval in ticks. Both are `NpgsqlDbType.Bigint`.
    - The result is exact because both sides are µs-aligned (R6), and it cannot overflow before year 9999.
  - **(as built)** The bucket timestamp returned for a Processed bucket is the *rounded* window start plus `i × interval`, so with a
    sub-µs request start it differs from SQL Server's label by the sub-µs remainder; values and membership are identical.
  - The query returns the full `BucketRow` statistics. Usable, good, bad and uncertain counts come from
    `sum(CASE …)` over the status column.
  - It does **not** use `SAMPLE BY`, whose alignment rules differ from ours. Computing the index ourselves keeps the
    bucket edges identical to SQL Server: start inclusive, end exclusive, a partial final bucket, interval 0, and the
    100k cap.
  - The time of the min or max value (ActualTime family) comes from a `row_number()` window function in the same single
    statement: `ORDER BY num_value ASC|DESC, ts ASC`, with ties going to the earliest sample (T0-confirmed; the form is
    below). No second statement is used. `AggregateEvaluator` then produces the values.
  - **Confirmed forms (T0 Q8).** They mirror `SqlAggregateQueries`'s three families and column order. They were checked
    against hand-computed seeds and a 60k-row randomized cross-check across 4 partitions: 617 buckets, 0 mismatches.
    - Classified subquery `s`:
      ```sql
      SELECT (cast(ts AS LONG) * 10 - $1) / $2 AS b, ts, kind, num_value, int_value, str_value,
             CASE WHEN status < 2147483648 AND kind <> 0 THEN 1 ELSE 0 END AS u,
             CASE WHEN status < 1073741824 AND kind <> 0 THEN 1 ELSE 0 END AS g
      FROM {p}sample WHERE tag = $3 AND ts >= $4 AND ts < $5 AND NOT deleted
      ```
    - Counts (ordinals 1–5 after `b`): `count() AS total_count, sum(u) AS usable_count, sum(g) AS good_count, count() -
      sum(u) AS bad_count, sum(CASE WHEN u = 1 AND kind IN (4, 5) THEN 1 ELSE 0 END) AS non_numeric_count`.
    - **Simple:** `SELECT b, <counts>, avg(CASE WHEN u = 1 THEN num_value END) AS avg_value, min(CASE WHEN u = 1 THEN
      num_value END) AS min_value, max(…) AS max_value, sum(…) AS sum_value FROM (<s>) GROUP BY b ORDER BY b`.
    - **StartEnd:**
      ```sql
      WITH s AS (<s>), st AS (SELECT b, <counts> FROM s GROUP BY b),
           f AS (SELECT b, kind, num_value, int_value, str_value, ts,
                        row_number() OVER (PARTITION BY b ORDER BY ts) AS rf,
                        row_number() OVER (PARTITION BY b ORDER BY ts DESC) AS rl FROM s WHERE u = 1)
      SELECT st.*, x.kind, x.num_value, x.int_value, x.str_value, x.ts, y.kind, y.num_value, y.int_value, y.str_value, y.ts
      FROM st LEFT JOIN (SELECT * FROM f WHERE rf = 1) x ON x.b = st.b
              LEFT JOIN (SELECT * FROM f WHERE rl = 1) y ON y.b = st.b ORDER BY st.b
      ```
    - **ActualTime:** the same `s`/`st`, then `r AS (SELECT b, num_value, ts, row_number() OVER (PARTITION BY b ORDER
      BY num_value ASC|DESC, ts ASC) AS rn FROM s WHERE u = 1 AND num_value IS NOT NULL) SELECT st.*, r.num_value, r.ts
      FROM st LEFT JOIN (SELECT * FROM r WHERE rn = 1) r ON r.b = st.b ORDER BY st.b`.
    - A bucket with no usable sample returns NULL times but **`kind = 0`** (BYTE has no NULL). The mapper keys absence on
      the NULL `ts`.
    - `first()/last()` and `arg_min/arg_max` also returned correct values, but no `ORDER BY` fixes their tie order. The
      window forms state the earliest-timestamp tie-break explicitly and are the ones to use.
- **Errors.** A failed read fails only that node, under the same data-source contract as SQL Server.

## 4. Configuration (`ServerHistorian:QuestDb`)

| Key | Default | Validation |
|---|---|---|
| `ConnectionString` | — | Required when `Backend=QuestDb` and a consumer is enabled. An Npgsql string. **T0-confirmed:** `Host=questdb;Port=8812;Username=admin;Password=quest;Database=qdb;ServerCompatibilityMode=NoTypeLoading` (the docker-dev overlay from the host: `Host=localhost;Port=18812;…`). `ServerCompatibilityMode=NoTypeLoading` is kept: it skips the `pg_type` load, and every confirmed form was run with it. The string also works without it. `No Reset On Close` is **not** needed, because pooled open → close → reopen works. |
| `TablePrefix` | `hist_` | `^[A-Za-z0-9_]+$`. Only checked when a consumer is enabled. |
| `RetentionDays` | unset | ≥ 1 when set |
| `CommandTimeout` | `00:00:30` | > 0 |
| `AutoCreateSchema` | `true` | — |
| `VisibilityTimeout` | `00:00:05` | ≥ 0. 0 disables the wait. |

The Gateway keys (`Endpoint`, `ApiKey`, `UseTls`) are not required for QuestDb, the same as for SqlServer.

**(as built) Pool size.** QuestDB's default limit is **64** PG-wire connections; Npgsql's default `Maximum Pool Size` is **100**, and the
read, write, update and alarm paths share one pool per node. Operators should set `Maximum Pool Size` in the connection string below the
server limit, per node (a redundant pair shares one QuestDB). The validator checks the prefix only when a consumer is enabled, but the
connection helper also refuses a malformed prefix (`ArgumentException`) when the first seam is built.

## 5. Testing

- **T1 refactor gate.** The SQL Server integration suite (118) and unit suite pass unchanged before and after the
  extraction. Each moved piece of logic is falsified once against its existing test.
- **Conformance suite** (`tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Conformance`, a class library of abstract
  xUnit tests). Each backend's integration project subclasses it and supplies a fixture and an `IConformanceSeeder`,
  which writes rows directly in that backend's schema.
  - It covers raw forward, reverse and paging; AtTime, including the outage, after-last and future rules; all 13
    aggregates with the existing hand-computed literals; the quality contract; insert-if-absent writes and
    redelivery; the HistoryUpdate result codes; and over-long tag names.
  - Each legitimate backend difference is a named, documented override, never a silent skip. Microsecond rounding is
    the only expected one.
- **QuestDB-only tests:**
  - a tombstone hides the row and a redelivery does not resurrect it;
  - the read-your-writes wait;
  - TTL is applied, verified through `tables()` metadata;
  - concurrent bootstrap from two instances;
  - intra-batch microsecond collisions;
  - the `hist_tag` upsert;
  - transient-failure behaviour, writing against a stopped port;
  - **(as built)** pre-1970 handling (writer drop, `BadOutOfRange`, read bounds), the `long.MinValue` read repair, Insert over a tombstone,
    per-tag updater serialization, and a forward cap-1 paging regression against a real QuestDB.
  - Conformance differences are named overrides: pre-1970 seeds are skipped on QuestDB, and AtTime timestamp assertions compare `Round(x)` to
    the requested time (identity on SQL Server). The QuestDB fixture prunes idle pool connections after 1 s because of the 64-connection limit.
- **Fixture.** `HISTORIAN_QUESTDB_TEST_CONNECTION`; when it is absent, the tests skip. Each test class uses a unique
  throwaway `TablePrefix` (`t_<guid>_`) and drops its tables on dispose.
- **Host.** Registry and validator tests for `Backend=QuestDb`, beside the SqlServer ones.

## 6. Environment

- QuestDB is an **optional overlay file**, `docker-dev/docker-compose.questdb.yml`, so the default rig is unchanged.
  It defines the `questdb` service:
  - image `questdb/questdb:10.0.1`, pinned by T0 and pushed 2026-08-24;
  - container `otopcua-dev-questdb-1`;
  - started with `docker compose -f docker-dev/docker-compose.yml -f docker-dev/docker-compose.questdb.yml up -d
    --no-deps questdb`.
- It exposes PG wire on host port **18812** (→ 8812) and the web console/REST on host port **19000** (→ 9000), so it
  does not collide with AdminUI `:9000`/`:9200`.
- It uses the named volume `otopcua-questdb-data`.
- The central nodes gain `ServerHistorian__QuestDb__ConnectionString` (`OTOPCUA_HISTORIAN_QUESTDB`), defaulting to the
  `questdb` service.
- No new NuGet packages.

## 7. Live gate (T9, run by the controller)

On docker-dev, with `Backend=QuestDb`:
1. Only the Primary records, with real kinds and status, one row per value.
2. Raw and Processed HistoryRead return identical results through the raw and UNS NodeIds, and match a direct
   QuestDB query.
3. HistoryUpdate covers insert, duplicate, replace, replace-missing and delete-at-time, each confirmed by a read and
   by a direct query. After a delete, an outbox redelivery does not bring the row back. A WriteOperate-only key and an
   anonymous session are both denied.
4. On failover, recording continues at 1×.
5. Modbus device loss stores a `BadCommunicationError` Empty sample, then the `BadNoCommunication` outage marker,
   then Good on recovery.
6. TTL is visible in table metadata when `RetentionDays` is set.

Evidence goes in `docs/plans/2026-10-10-questdb-historian-live-gate.md`, and the rig is restored afterwards.

## 8. Work order

| Wave | Task | Model |
|---|---|---|
| W0 | **T0 spike.** Pin the image, add the compose service, and prove every QuestDB assumption here: Npgsql parameterised multi-row inserts, `DEDUP UPSERT`, `wal_tables()` visibility, TTL syntax, microsecond timestamps over Npgsql, the min/max-time query form, AtTime neighbour lookup, concurrent `CREATE TABLE IF NOT EXISTS`. **Amend §2–§4 with the findings before W1.** | controller + sonnet |
| W1 | T1: extract `Sql.Common`, leaving SQL Server's behaviour unchanged | opus |
| W2 | T2 conformance suite over SQL Server ‖ T3 QuestDB skeleton (options, connection, bootstrapper, fixture) | sonnet ‖ sonnet |
| W3 | T4 writes, provisioning and alarms ‖ T5 reads and aggregates ‖ T6 updater with soft delete | sonnet ‖ opus ‖ sonnet |
| W4 | T7 provider, registry and validator, plus the conformance suite run against QuestDB | sonnet |
| W5 | T8 docs (Historian.md, Configuration.md, CLAUDE.md, docker-dev README), then a Fable whole-branch review, one fix wave, and T9 live gate | sonnet / fable / opus / controller |