dohertj2/lmxopcua
1
0
Fork 0
Code Issues 14 Pull Requests Actions Packages Projects Releases Wiki Activity

spike(historian): QuestDB behaviours confirmed against pinned image; spec §2–§4 amended

Browse Source 
questdb/questdb:10.0.1 (pushed 2026-08-24). Q1–Q13 answered with harness output in
docs/plans/2026-10-10-questdb-spike.md; confirmed DDL, visibility wait, bucket/neighbour
SQL and connection keywords folded into the design spec, marked T0-confirmed.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
This commit is contained in:
Joseph Doherty
2026-10-11 01:07:38 -04:00
parent f54ab04e64
commit cdea173bd1
3 changed files with 608 additions and 19 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docker-dev/docker-compose.questdb.yml
+28
View File
@@ -0,0 +1,28 @@
# docker-dev/docker-compose.questdb.yml — optional QuestDB fixture for the QuestDB historian backend (#666).
#
# Up (touches ONLY the questdb service; it has no depends_on, and --no-deps makes that explicit):
# docker compose -f docker-dev/docker-compose.yml -f docker-dev/docker-compose.questdb.yml up -d --no-deps questdb
#
# Image pinned by the T0 spike (docs/plans/2026-10-10-questdb-spike.md): questdb/questdb:10.0.1,
# pushed to Docker Hub 2026-08-24 (≥ 2 weeks before 2026-10-10; ≥ 8.2.3, the first release with table TTL).
# The image ships /usr/bin/curl, so the healthcheck uses it against the min-HTTP health endpoint on :9003.
#
# PG wire -> localhost:18812 (Host=localhost;Port=18812;Username=admin;Password=quest;Database=qdb;...)
# Console -> http://localhost:19000
services:
questdb:
image: questdb/questdb:10.0.1
container_name: otopcua-dev-questdb-1
ports:
- "18812:8812" # PG wire
- "19000:9000" # web console / REST
volumes:
- otopcua-questdb-data:/var/lib/questdb
healthcheck:
test: ["CMD-SHELL", "curl -fs http://localhost:9003/status || exit 1"]
interval: 10s
timeout: 5s
retries: 12
volumes:
otopcua-questdb-data:
name: otopcua-questdb-data
docs/plans/2026-10-10-questdb-historian-design.md
+169 -19
View File
@@ -81,28 +81,64 @@ Host changes:
QuestDB has no schemas, so the tables carry a configurable `TablePrefix` (default `hist_`, `[A-Za-z0-9_]+`). All
three are **WAL tables with `DEDUP UPSERT KEYS`**, so re-sending a row with the same key replaces it.
**T0-confirmed** against `questdb/questdb:10.0.1` (`docs/plans/2026-10-10-questdb-spike.md` §DDL, Q3, Q5, Q10). The
`TTL n DAYS` clause goes **between `PARTITION BY` and `WAL`**. Placed after the dedup clause it fails with `unexpected
token [TTL]`. It is emitted only when `RetentionDays` is set, as an inlined integer, because it cannot be a parameter.
```sql
CREATE TABLE IF NOT EXISTS hist_sample (
ts TIMESTAMP, tag SYMBOL, kind BYTE, num_value DOUBLE, int_value LONG, str_value VARCHAR,
status LONG, deleted BOOLEAN, server_ts TIMESTAMP
) TIMESTAMP(ts) PARTITION BY DAY WAL DEDUP UPSERT KEYS(ts, tag) [TTL n DAYS];
) TIMESTAMP(ts) PARTITION BY DAY [TTL n DAYS] WAL DEDUP UPSERT KEYS(ts, tag);
CREATE TABLE IF NOT EXISTS hist_event (
ts TIMESTAMP, event_id VARCHAR, source SYMBOL,
-- plus the remaining hist.Event columns one-for-one (docs/historian/sqlserver/001_initial.sql), QuestDB-typed
) TIMESTAMP(ts) PARTITION BY DAY WAL DEDUP UPSERT KEYS(ts, event_id) [TTL n DAYS];
ts TIMESTAMP, event_id VARCHAR, source SYMBOL, alarm_id VARCHAR, alarm_name VARCHAR, alarm_type_name VARCHAR,
severity INT, event_kind VARCHAR, message VARCHAR, user_name VARCHAR, comment VARCHAR, received_utc TIMESTAMP
) TIMESTAMP(ts) PARTITION BY DAY [TTL n DAYS] WAL DEDUP UPSERT KEYS(ts, event_id);
CREATE TABLE IF NOT EXISTS hist_tag (
ts TIMESTAMP, name SYMBOL, data_type BYTE, eng_unit VARCHAR, description VARCHAR, updated_utc TIMESTAMP
) TIMESTAMP(ts) PARTITION BY YEAR WAL DEDUP UPSERT KEYS(ts, name); -- ts is always 1970-01-01, which makes this an upsert by name
```
Column details:
- `status` holds the uint status code as a LONG.
- `int` also holds Booleans as 0/1, matching SQL Server.
- `kind` is `HistorianValueKind`.
- The exact column types and the TTL syntax are confirmed by T0 against the pinned image. T0 amends this section
with what it finds.
`hist_event` maps `hist.Event` one-for-one:
- `EventTimeUtc` → `ts`, `EventId` → `event_id`, `SourceName` → `source`;
- `AlarmId`/`AlarmName`/`AlarmTypeName` → `alarm_id`/`alarm_name`/`alarm_type_name`;
- `Severity` → `severity`, `EventKind` → `event_kind`, `Message` → `message`;
- `UserName` → `user_name`, `Comment` → `comment`, `ReceivedTimeUtc` → `received_utc`.
`event_id VARCHAR` works as a dedup key (T0 Q3), so re-sending an event replaces it. The S4 fallback (no dedup on
`hist_event`) is not used.
Column details (**T0-confirmed**):
- `status` holds the uint status code **unsigned** in a LONG (0..4294967295, never negative). So "severity Bad" is
`status >= 2147483648`, not `< 0` as in SQL Server's signed `int`.
- `int_value` also holds Booleans as 0/1, matching SQL Server.
- `kind` is `HistorianValueKind` in a `BYTE`.
- Read-back CLR types:
| Column type | Npgsql type name | CLR type / reader call |
|---|---|---|
| `BYTE` | `smallint` | `Int16` (`GetInt16`) |
| `SYMBOL`, `VARCHAR` | `character varying` | `String` |
| `LONG` | `bigint` | `Int64` |
| `INT` | `integer` | `Int32` |
| `BOOLEAN` | `boolean` | `Boolean` |
| `DOUBLE` | `double precision` | `Double` |
| `TIMESTAMP` | `timestamp without time zone` | `DateTime`, **`Kind=Unspecified`**; the reader applies `SpecifyKind(…, Utc)` |
- **NULL semantics:**
- DOUBLE NaN and ±Inf read back as NULL.
- `long.MinValue` (LONG) and `int.MinValue` (INT) are QuestDB's NULL sentinels.
- `BYTE` and `BOOLEAN` have **no NULL**: unset or DBNull reads `0` / `false`. So `deleted` defaults to `false`, and a
LEFT-JOINed "no row" yields `kind = 0`. Absence must be detected on a NULL `ts`, not on `kind`.
- **Ruling candidate (T0):** an Int64 sample of exactly `long.MinValue` stores `int_value` as NULL. The QuestDB read
path repairs it before the shared `FromColumns`: `kind = Int64 ∧ int_value IS NULL ∧ num_value = -2^63` ⇒
`long.MinValue`. That double is exact. Without the repair the sample would be served as 0.
- A designated timestamp **before 1970-01-01 cannot be inserted** (`designated timestamp before 1970-01-01 is not
allowed`). See §2.2.
- Tag names: SYMBOL and VARCHAR take 400 characters (non-ASCII included). SYMBOL grows past its capacity hint
(16,384) with no error, and was tested with 30,000 distinct tags. No `CAPACITY` clause is needed.
### 2.2 Live writes (recorder) — insert only if absent, the same as SQL Server
@@ -110,16 +146,45 @@ Column details:
to microseconds before this step, so microsecond collisions also collapse.
2. For each tag, fetch the timestamps that already exist in the batch's `[min ts, max ts]` range. **Tombstones are
included**, so a re-sent outbox batch can never resurrect a deleted row or overwrite a HistoryUpdate change.
3. Insert only the new rows, in one parameterised multi-row `INSERT` per tag group.
T0-confirmed form (Q11): `SELECT ts FROM {p}sample WHERE tag = $1 AND ts >= $2 AND ts <= $3`, with no `NOT
deleted`, followed by a client-side set test.
3. Insert only the new rows with parameterised multi-row `INSERT … VALUES (…),(…)`, **at most 1,000 rows per
statement** (T0-confirmed Q2). QuestDB reads the Bind parameter count as a signed int16, so the hard limit is
32,767 parameters (3,640 nine-column rows). Within one statement a duplicate key keeps the **last** row, which is
why step 1's "first wins" is done client-side.
4. Stored-value rules are shared with SQL Server:
- NaN/±Inf, or any Empty with a non-Bad status, is stored Empty with `BadOutOfRange` or `BadNoData`.
- Comms-loss outage markers are stored as given.
- A tag name over 400 characters is a per-tag failure.
- **Ruling candidate (T0 Q13):** a sample timestamped before 1970-01-01 cannot be stored, because QuestDB rejects
the whole INSERT. The writer drops such samples before the INSERT and logs one Warning per batch. A retry could
never succeed, and sending the sample would push the tag's whole group into the #538 quarantine.
5. Any SQL or transport failure makes the write return `false`, so the recorder retries and the #538 per-tag
quarantine still applies. It never throws.
The recorder is Primary-only (#537), so step 2 does not race another live writer. The recorder waits for WAL
visibility, bounded by `VisibilityTimeout` (plan R5).
visibility, bounded by `VisibilityTimeout` (plan R5), using the §2.3 wait.
Binding (T0-confirmed Q1/Q6/Q13):
- Every timestamp parameter is `NpgsqlDbType.TimestampTz` with `DateTimeKind.Utc`, after `Round`. `Timestamp` + `Utc`
throws in Npgsql. Npgsql truncates sub-µs ticks **toward 2000-01-01**, which rounds *up* before 2000. A raw
`DateTime.MinValue`/`MaxValue` is sent as ±infinity, and a `[Min, Max)` range then silently matches nothing. R6's
`BindTimestamp` (round; `MinValue` → `+10` ticks; `MaxValue` → `−9` ticks) fixes all three.
- The other parameter types:
| Column | `NpgsqlDbType` |
|---|---|
| `tag`/`str_value`/other strings | `Varchar` |
| `kind`/`data_type` | `Smallint` |
| `num_value` | `Double` |
| `int_value`/`status` | `Bigint` |
| `deleted` | `Boolean` |
| `severity` | `Integer` |
NULL is `DBNull.Value` with the column's type.
- Every QuestDB server error carries **SQLSTATE `00000`**. `QuestDbConnection`'s classification therefore treats a
`PostgresException` as a server-side rejection, judged by `MessageText`. Any other `NpgsqlException`, `IOException`,
`SocketException` or timeout is transport (transient).
### 2.3 HistoryUpdate (soft delete)
@@ -140,6 +205,25 @@ Further rules:
- **Read-your-writes.** After its inserts, the updater polls `wal_tables()` until the table's applied transaction
reaches the sequencer transaction it produced, up to `VisibilityTimeout`. On a timeout the write is kept, the codes
are reported as normal, and a Warning is logged.
- **The visibility wait (`QuestDbWalVisibility`, T0-confirmed Q4)** is used by the writer and the updater (R5):
1. After the INSERT(s) return: `SELECT sequencerTxn FROM wal_tables() WHERE name = $1` (`$1` = the full table name,
`Varchar`) → `target`. Each autocommitted INSERT is exactly one sequencer transaction.
2. Poll `SELECT writerTxn, suspended FROM wal_tables() WHERE name = $1` about every 10 ms until `writerTxn >=
target`. Stop early, with a Warning, if `suspended` is true. Stop at `VisibilityTimeout`.
- Measured: median 0 ms, p95 6 ms, max 13 ms. Up to 7/20 inserts were not visible to an immediate re-read, so the
wait is required. A concurrent writer can only raise `target`. The key-poll fallback (S11) is not used.
- Existence checks for sparse item times (Insert/Replace/Update/DeleteAtTime) use `SELECT ts, deleted FROM {p}sample
WHERE tag = $1 AND ts IN ($2, …)`, chunked at 1,000 times (T0 Q11).
- **DeleteRaw (T0-confirmed), page size 1,000:**
1. `SELECT ts FROM {p}sample WHERE tag = $1 AND ts >= $2 AND ts < $3 AND NOT deleted ORDER BY ts LIMIT 1000`.
2. Write one 1,000-row tombstone INSERT.
3. Run the visibility wait.
4. Continue from `last ts + 10 ticks`, until a page comes back empty.
The wait between pages is required, or the next page re-reads rows just tombstoned. Measured: 2,500 rows, 3 pages,
103 ms.
- **Ruling candidate (T0 Q13):** an Insert, Replace or Update item timestamped before 1970-01-01 returns
`BadOutOfRange`, because QuestDB cannot store it. Deletes there find nothing and return `BadNoEntryExists`. Pre-1970
conformance seeds are skipped for QuestDB.
- The existence checks for Insert and Replace read committed data. If a live recorder sample at the *same*
timestamp is still in flight in the WAL, it can race the updater. Updates normally target past timestamps, so this
is accepted and documented.
@@ -150,20 +234,56 @@ When `RetentionDays` is set, the bootstrapper applies TTL to `hist_sample` and `
the configured value changes. QuestDB then drops expired day partitions, tombstones included. When unset, data is
kept forever. `hist_tag` never expires.
T0-confirmed (Q5):
- **Create:** `… PARTITION BY DAY TTL n DAYS WAL …` (§2.1).
- **Change:** `ALTER TABLE {p}sample SET TTL n DAYS`. **Remove** (`RetentionDays` unset while the table has a TTL):
`ALTER TABLE … SET TTL 0 DAYS`.
- `n` cannot be a `$` parameter (`invalid syntax, should be <number> <unit> but was $1`). It is the validated integer,
formatted with `InvariantCulture`.
- TTL is not allowed on the `PARTITION BY YEAR` `hist_tag` table (`TTL value must be an integer multiple of the
partition size`).
- **Reading the current TTL:** `SELECT ttlValue, ttlUnit FROM tables() WHERE table_name = $1`. QuestDB **normalises
the unit**: 7 days reads `1 WEEK`, 14 days `2 WEEK`, 30 days `30 DAY`, none `0 HOUR`. The bootstrapper compares in
hours (`HOUR`=1, `DAY`=24, `WEEK`=168; any other unit counts as different) and issues an `ALTER` only on a
difference.
- `ALTER … SET TTL` is applied through the WAL, so `tables()` shows the old value for 25–100 ms. The bootstrapper does
not re-read to verify.
- Expiry is checked against the **wall clock**: the image default is `cairo.ttl.use.wall.clock=true`. A future-dated
sample does not expire current data (T0 Q5c).
- **Table listing:** `SELECT table_name FROM tables() WHERE starts_with(table_name, $1)`. Never `LIKE 'hist_%'`,
because `_` is a wildcard.
- **Concurrent bootstrap:** `CREATE TABLE IF NOT EXISTS` from 4 connections × 20 rounds produced no errors. The S11
defence stays: catch a `PostgresException` whose `MessageText` contains `already exists`, re-check `tables()`, and
rethrow if the table is absent. Never key on SQLSTATE (always `00000`).
## 3. Reads
Every query filters `NOT deleted`. Every value goes through `StoredSample`, so the quality clamp is the same as SQL
Server's.
- **Raw.** A range on `ts`, `ORDER BY ts [DESC]`, `LIMIT @top` (`maxValuesPerNode == 0` ⇒ unbounded), at most `top` samples, `ContinuationPoint: null`; the node manager pages by re-reading from the last timestamp (plan R8). The dedup key forbids same-instant ties. Bounds follow the existing data-source contract.
- **AtTime.** Find the previous-or-equal neighbour and the next neighbour of each requested time. The default is
`LIMIT 1` seeks on `ts`, batched into chunked statements; T0 picks `ASOF JOIN` for the previous neighbour if it
measures better. `AtTimeEvaluator` then decides the result.
- **AtTime (T0-confirmed Q9).** Find the previous-or-equal neighbour and the next neighbour of each requested time
with `LIMIT 1` seeks, `UNION ALL`-ed, **at most 50 request times (100 branches) per statement**. QuestDB rejects
128 or more branches with `SQL model is too complex to evaluate`. Per time `k`, with `$1` = tag and `$j` = the
rounded time:
```sql
SELECT * FROM (SELECT k AS req, 0 AS dir, ts, kind, num_value, int_value, str_value, status FROM {p}sample
WHERE tag = $1 AND ts <= $j AND NOT deleted ORDER BY ts DESC LIMIT 1)
UNION ALL
SELECT * FROM (SELECT k AS req, 1 AS dir, ts, kind, num_value, int_value, str_value, status FROM {p}sample
WHERE tag = $1 AND ts > $j AND NOT deleted ORDER BY ts ASC LIMIT 1)
```
`k` and `dir` are generated integer literals. Measured: 102 times in about 16–20 ms over 100k rows per tag, with no
mismatches. `ASOF JOIN` was faster (1 ms), but it yields only the previous neighbour, so it was not chosen.
`AtTimeEvaluator` then decides the result.
- **Events.** Filtered by source and `ts` range, ordered ascending (reversed windows are swapped, as built today),
and limited to `maxEvents`.
- **Processed.** **One query per node.**
- Bucket index: `(cast(ts AS LONG) - @startMicros) / @intervalMicros`, using microsecond values from `BucketPlan`,
grouped by the index.
- **Processed.** **One query per node** (T0-confirmed for every family; the S11 two-statement fallback is not needed).
- Bucket index (T0-confirmed): `(cast(ts AS LONG) * 10 - $1) / $2`.
- `cast(ts AS LONG)` is µs since 1970.
- `$1` = the rounded window start in **ticks since the Unix epoch** (`start.Ticks − DateTime.UnixEpoch.Ticks`).
- `$2` = the interval in ticks. Both are `NpgsqlDbType.Bigint`.
- The result is exact because both sides are µs-aligned (R6), and it cannot overflow before year 9999.
- The query returns the full `BucketRow` statistics. Usable, good, bad and uncertain counts come from
`sum(CASE …)` over the status column.
- It does **not** use `SAMPLE BY`, whose alignment rules differ from ours. Computing the index ourselves keeps the
@@ -171,13 +291,43 @@ Server's.
100k cap.
- The time of the min or max value needs an arg-min style lookup. It is computed with a window function, or with a
second statement if QuestDB lacks one; T0 decides. `AggregateEvaluator` then produces the values.
- **Confirmed forms (T0 Q8).** They mirror `SqlAggregateQueries`'s three families and column order. They were checked
against hand-computed seeds and a 60k-row randomized cross-check across 4 partitions: 617 buckets, 0 mismatches.
- Classified subquery `s`:
```sql
SELECT (cast(ts AS LONG) * 10 - $1) / $2 AS b, ts, kind, num_value, int_value, str_value,
CASE WHEN status < 2147483648 AND kind <> 0 THEN 1 ELSE 0 END AS u,
CASE WHEN status < 1073741824 AND kind <> 0 THEN 1 ELSE 0 END AS g
FROM {p}sample WHERE tag = $3 AND ts >= $4 AND ts < $5 AND NOT deleted
```
- Counts (ordinals 1–5 after `b`): `count() AS total_count, sum(u) AS usable_count, sum(g) AS good_count, count() -
sum(u) AS bad_count, sum(CASE WHEN u = 1 AND kind IN (4, 5) THEN 1 ELSE 0 END) AS non_numeric_count`.
- **Simple:** `SELECT b, <counts>, avg(CASE WHEN u = 1 THEN num_value END) AS avg_value, min(CASE WHEN u = 1 THEN
num_value END) AS min_value, max(…) AS max_value, sum(…) AS sum_value FROM (<s>) GROUP BY b ORDER BY b`.
- **StartEnd:**
```sql
WITH s AS (<s>), st AS (SELECT b, <counts> FROM s GROUP BY b),
f AS (SELECT b, kind, num_value, int_value, str_value, ts,
row_number() OVER (PARTITION BY b ORDER BY ts) AS rf,
row_number() OVER (PARTITION BY b ORDER BY ts DESC) AS rl FROM s WHERE u = 1)
SELECT st.*, x.kind, x.num_value, x.int_value, x.str_value, x.ts, y.kind, y.num_value, y.int_value, y.str_value, y.ts
FROM st LEFT JOIN (SELECT * FROM f WHERE rf = 1) x ON x.b = st.b
LEFT JOIN (SELECT * FROM f WHERE rl = 1) y ON y.b = st.b ORDER BY st.b
```
- **ActualTime:** the same `s`/`st`, then `r AS (SELECT b, num_value, ts, row_number() OVER (PARTITION BY b ORDER
BY num_value ASC|DESC, ts ASC) AS rn FROM s WHERE u = 1 AND num_value IS NOT NULL) SELECT st.*, r.num_value, r.ts
FROM st LEFT JOIN (SELECT * FROM r WHERE rn = 1) r ON r.b = st.b ORDER BY st.b`.
- A bucket with no usable sample returns NULL times but **`kind = 0`** (BYTE has no NULL). The mapper keys absence on
the NULL `ts`.
- `first()/last()` and `arg_min/arg_max` also returned correct values, but no `ORDER BY` fixes their tie order. The
window forms state the earliest-timestamp tie-break explicitly and are the ones to use.
- **Errors.** A failed read fails only that node, under the same data-source contract as SQL Server.
## 4. Configuration (`ServerHistorian:QuestDb`)
| Key | Default | Validation |
|---|---|---|
| `ConnectionString` | — | Required when `Backend=QuestDb` and a consumer is enabled. An Npgsql string, e.g. `Host=questdb;Port=8812;Username=admin;Password=quest;Database=qdb;ServerCompatibilityMode=NoTypeLoading` (T0 confirms the exact keywords). |
| `ConnectionString` | — | Required when `Backend=QuestDb` and a consumer is enabled. An Npgsql string. **T0-confirmed:** `Host=questdb;Port=8812;Username=admin;Password=quest;Database=qdb;ServerCompatibilityMode=NoTypeLoading` (the docker-dev overlay from the host: `Host=localhost;Port=18812;…`). `ServerCompatibilityMode=NoTypeLoading` is kept: it skips the `pg_type` load, and every confirmed form was run with it. The string also works without it. `No Reset On Close` is **not** needed, because pooled open → close → reopen works. |
| `TablePrefix` | `hist_` | `^[A-Za-z0-9_]+$`. Only checked when a consumer is enabled. |
| `RetentionDays` | unset | ≥ 1 when set |
| `CommandTimeout` | `00:00:30` | > 0 |
docs/plans/2026-10-10-questdb-spike.md
+411
View File
@@ -0,0 +1,411 @@
# QuestDB historian backend — T0 feasibility spike
**Status:** run 2026-10-11 against a live container. Gitea #666. Spec: `docs/plans/2026-10-10-questdb-historian-design.md`
(§2–§4 amended in place from this doc; every confirmed form there is marked **T0-confirmed**).
## Fixture
| Item | Value |
|---|---|
| Image | **`questdb/questdb:10.0.1`** — Docker Hub `tag_last_pushed` **2026-08-24T14:00:08Z** (48 days before 2026-10-10) |
| Why that tag | Newest exact `x.y.z` tag ≥ 14 days older than 2026-10-10. Newer tags are `nightly*` / `patch-*` only. Floor for TTL is **8.2.3**; 10.0.1 is well above it. |
| Build | `SELECT build()` → `Build Information: QuestDB 10.0.1, JDK 25.0.2, Commit Hash 7a391566ac827e0d8c269b91f2415c16b0a32a84` |
| Overlay | `docker-dev/docker-compose.questdb.yml` — container `otopcua-dev-questdb-1`, ports `18812:8812` (PG wire) and `19000:9000` (console/REST), named volume `otopcua-questdb-data` (explicit `name:` so it is not project-prefixed) |
| Up | `docker compose -f docker-dev/docker-compose.yml -f docker-dev/docker-compose.questdb.yml up -d --no-deps questdb` (a `--dry-run` showed only the volume + `otopcua-dev-questdb-1` being created; nothing else in the `otopcua-dev` project is touched) |
| Healthcheck | the image ships `/usr/bin/curl` (`command -v curl` → `/usr/bin/curl`; no `wget`), so `curl -fs http://localhost:9003/status` is kept. Container reached `healthy`. |
| Harness | throwaway net10.0 console app, Npgsql **10.0.3** (the repo's central pin, `Directory.Packages.props` line 97), in the session scratchpad `questdb-spike/` with its own `Directory.Build.props`, `Directory.Packages.props` and `nuget.config`. Nothing was added to the repo. |
### Connection string (confirmed)
```
Host=localhost;Port=18812;Username=admin;Password=quest;Database=qdb;ServerCompatibilityMode=NoTypeLoading
```
Inside the docker network the same string uses `Host=questdb;Port=8812`.
## Q1 — keywords, parameter types, CLR types
```
[OK] open+SELECT 1 with 'Host=localhost;Port=18812;Username=admin;Password=quest;Database=qdb'
[OK] open+SELECT 1 with '...;ServerCompatibilityMode=NoTypeLoading'
[OK] open->close->reopen x5 on one data source (no 'No Reset On Close')
[OK] reopen after a parameterised INSERT (prepared-ish state)
[FAIL] ts Utc + Timestamp: ArgumentException: Cannot write DateTime with Kind=UTC to PostgreSQL type 'timestamp without time zone', consider using 'timestamp with time zone'. ...
[OK] ts Utc + TimestampTz
[OK] ts Utc + inferred
[OK] ts Unspecified + Timestamp
[FAIL] ts Local + TimestampTz: ArgumentException: Cannot write DateTime with Kind=Local to PostgreSQL type 'timestamp with time zone', only UTC is supported. ...
[OK] kind bound as Integer (int4)
[OK] tag bound as Text
[OK] NaN double insert
[OK] +Inf double insert
cols: ts:timestamp without time zone/DateTime, tag:character varying/String, kind:smallint/Int16, num_value:double precision/Double,
int_value:bigint/Int64, str_value:character varying/String, status:bigint/Int64, deleted:boolean/Boolean, server_ts:timestamp without time zone/DateTime
2026-01-02T03:04:05.1234560(Unspecified) | ts Utc + TimestampTz | 3<Int16> | 1.5 | 7<Int64> | s | 2147483648<Int64> | False | 2026-10-11T04:56:27.1103710(Unspecified)
...
2026-01-02T03:04:11.1234560(Unspecified) | nan | 0<Int16> | NULL | NULL | NULL | NULL | False | NULL
2026-01-02T03:04:12.1234560(Unspecified) | inf | 0<Int16> | NULL | NULL | NULL | NULL | False | NULL
integer arithmetic with $n:
cols: ts::LONG - $1:bigint/Int64 [OK] cast(ts AS LONG) - $1 (Bigint)
cols: (ts::LONG * 10 - $1) / $2:bigint/Int64 [OK] (Bigint) [OK] (Integer)
```
Sentinel NULLs (extra check):
```
cols: ts, l:bigint/Int64, i:integer/Int32, b:smallint/Int16, z:boolean/Boolean, l = NULL:boolean
2026-03-01T00:00:00 | NULL | NULL | 0 | False | True <- l = long.MinValue, i = int.MinValue written
2026-03-01T00:00:01 | -9223372036854775807 | -2147483647 | 0 | False | False
2026-03-01T00:00:02 | NULL | NULL | 0 | False | True <- BYTE/BOOLEAN bound as DBNull
```
**Decisions**
- Both connection strings work. **Keep `ServerCompatibilityMode=NoTypeLoading`**: it skips the `pg_type` load, and every
form in this doc was run with it on. **`No Reset On Close` is not needed**: open → close → reopen on one pooled
`NpgsqlDataSource` works.
- **Timestamps are bound as `NpgsqlDbType.TimestampTz` with `DateTimeKind.Utc`.** `Timestamp` + `Utc` throws in Npgsql
before reaching the server. `Unspecified` + `Timestamp` also works, but one rule is enough.
- **Timestamps read back as `DateTime` with `Kind=Unspecified`**, wall clock = UTC. The reader must
`DateTime.SpecifyKind(r.GetDateTime(i), DateTimeKind.Utc)`.
- CLR types returned: `BYTE` → `smallint` / **`Int16`** (`GetInt16`); `SYMBOL` → `character varying` / `String`;
`VARCHAR` → `String`; `LONG` → `Int64`; `INT` → `Int32`; `BOOLEAN` → `Boolean`; `TIMESTAMP` → `DateTime`;
`DOUBLE` → `Double`. `count()`/`sum(int)` → `Int64`.
- Parameter types: `BYTE` ← `Smallint` (`Integer` also accepted); `SYMBOL`/`VARCHAR` ← `Varchar` (`Text` also accepted);
`LONG` ← `Bigint`; `BOOLEAN` ← `Boolean`; `DOUBLE` ← `Double`; a NULL is `DBNull.Value` with the column's
`NpgsqlDbType`.
- `$n` in integer arithmetic: **bind as `NpgsqlDbType.Bigint`** (Integer also worked for small values; Bigint is the
rule because tick offsets exceed int32).
- **DOUBLE NaN and ±Inf read back as NULL.** **`LONG` `long.MinValue` and `INT` `int.MinValue` are QuestDB's NULL
sentinels** and read back NULL. **`BYTE` and `BOOLEAN` have no NULL**: unset or DBNull reads `0` / `false`. So
`deleted` defaults to `false` and `kind` of a never-written field is `0` (Empty).
- **Every server error carries `SqlState = 00000`.** Classification cannot use SQLSTATE: a `PostgresException` is a
server-side rejection (decide by `MessageText`), any other `NpgsqlException`/`IOException`/`SocketException`/timeout
is transport.
## Q2 — multi-row parameterised INSERT and batch size
```
[OK] 1000 rows x 9 params = 9000 params 88 ms
[OK] 2000 rows x 9 params = 18000 params 167 ms
[OK] 3640 rows x 9 params = 32760 params 342 ms
[FAIL] 3641 rows x 9 params = 32769 params: PostgresException: 00000: invalid parameter count [parameterCount=-32767, offset=225670
[FAIL] 5000 rows x 9 params = 45000 params: PostgresException: 00000: invalid parameter count [parameterCount=-20536, ...
[FAIL] 7282 rows x 9 params = 65538 params: NpgsqlException: A statement cannot have more than 65535 parameters
20 x 1000-row statements: fails=0, 122 ms total
count()=27251 | count_distinct(ts)=27251 expected rows = 27251
```
**Decision:** QuestDB reads the Bind parameter count as a **signed int16**, so the real limit is **32,767 parameters per
statement**, not Npgsql's 65,535. **Batch size = 1,000 rows per `INSERT … VALUES (…),(…)`** (9,000 parameters for
`hist_sample`), which is reliable and far under the limit. **DeleteRaw page size = 1,000 rows** (see the DDL section).
## Q3 — `DEDUP UPSERT KEYS`
```
before: 2026-03-01T00:00:00 | a | 1 | False 2026-03-01T00:00:00 | b | 2 | False
after re-insert of (t0,'a') as tombstone:
2026-03-01T00:00:00 | a | NULL | True 2026-03-01T00:00:00 | b | 2 | False
NOT deleted: 2026-03-01T00:00:00 | b | 2
same key twice in one statement (10 then 20): 2026-03-01T00:00:01 | c | 20
hist_event with event_id VARCHAR as a dedup key:
[OK] CREATE hist_event-shape with DEDUP UPSERT KEYS(ts, event_id) on VARCHAR
[OK] insert e1 'first', e2, then e1 'second'
2026-03-01T00:00:00 | e1 | src1 | 500 | second | NULL
2026-03-01T00:00:00 | e2 | src1 | 500 | other | NULL
```
**Decision:** re-inserting a key replaces the whole row; a tombstone written at `(ts, tag)` hides the row from `NOT
deleted`. **Within one statement the LAST duplicate wins**, so the writer's "first value wins" rule must be applied
client-side before the INSERT (spec §2.2 step 1 already does this). **`event_id VARCHAR` works as a dedup key**, so the
S4 fallback (no dedup on `hist_event`) is **not needed**.
## Q4 — `wal_tables()` and the visibility wait
```
cols: name:varchar, suspended:boolean, writerTxn:bigint, bufferedTxnSize:bigint, sequencerTxn:bigint,
errorTag:varchar, errorMessage:varchar, memoryPressure:integer
run 0: seqTxn before=0 after=1 visibleImmediately=False waitMs=13 visibleAfterWait=True
run 1: seqTxn before=1 after=2 visibleImmediately=False waitMs=0 visibleAfterWait=True
...
run 16: seqTxn before=16 after=17 visibleImmediately=False waitMs=6 visibleAfterWait=True
lag ms: min=0 median=0 p95=6 max=13; visible-without-wait 17/20
```
(An earlier run: visible without waiting 13/20, max 6 ms.) Every autocommitted INSERT is exactly one sequencer
transaction.
**Decision (the form T3's `QuestDbWalVisibility` implements):**
1. After the INSERT(s) return, on the same or any connection:
`SELECT sequencerTxn FROM wal_tables() WHERE name = $1` (`$1` = the table name, `Varchar`) → `target`.
2. Poll `SELECT writerTxn, suspended FROM wal_tables() WHERE name = $1` every ~10 ms until `writerTxn >= target`, or
`suspended` is true, or `VisibilityTimeout` elapses.
A concurrent writer can only make `target` larger, which lengthens the wait slightly and never shortens it. The wait is
needed: up to 7/20 inserts were not yet visible on the next query. The S11 key-poll fallback is **not needed**.
## Q5 — TTL
```
[FAIL] CREATE ... PARTITION BY DAY WAL DEDUP UPSERT KEYS(ts, tag) TTL 7 DAYS (TTL last): unexpected token [TTL]
[OK] CREATE ... PARTITION BY DAY TTL 7 DAYS WAL DEDUP UPSERT KEYS(ts, tag) (TTL before WAL)
spike_q5 | DAY | True | True | ttlValue=1 | ttlUnit=WEEK
[FAIL] ALTER TABLE spike_q5 SET TTL $1 DAYS (parameter): invalid syntax, should be <number> <unit> but was $1
[FAIL] TTL 7 DAYS on PARTITION BY YEAR table: TTL value must be an integer multiple of the partition size (its time interval)
Q5b ALTER ... SET TTL visibility in tables()
SET TTL 30 DAYS : seqTxn 6->7; applied after 100 ms; tables() now ttlValue=30 ttlUnit=DAY
SET TTL 14 DAYS : seqTxn 7->8; applied after 94 ms; tables() now ttlValue=2 ttlUnit=WEEK
SET TTL 1 DAY : seqTxn 8->9; applied after 59 ms; tables() now ttlValue=1 ttlUnit=DAY
SET TTL 0 DAYS : seqTxn 9->10; applied after 25 ms; tables() now ttlValue=0 ttlUnit=HOUR
SET TTL 3 DAYS : seqTxn 10->11; applied after 60 ms; tables() now ttlValue=3 ttlUnit=DAY
after inserting now-20d and now into a TTL 7 DAYS table:
new | 2026-10-11T05:03:03
Q5c TTL reference clock (future-dated row):
after inserting a now+30d row, then one more ordinary commit:
d-3 | 2026-10-08 now | 2026-10-11 now+1h | 2026-10-11 d+30 | 2026-11-10
```
Image `conf/server.conf`: `#cairo.ttl.use.wall.clock=true` — "when true (default), TTL enforcement uses wall clock
time to prevent accidental data loss when future timestamps are inserted".
**Decision:**
- Create syntax: **`… TIMESTAMP(ts) PARTITION BY DAY TTL n DAYS WAL DEDUP UPSERT KEYS(…)`** — TTL goes **between
`PARTITION BY` and `WAL`**. Alter: **`ALTER TABLE t SET TTL n DAYS`**; remove: **`ALTER TABLE t SET TTL 0 DAYS`**.
`n` cannot be a parameter; it is the validated integer `RetentionDays`, formatted with `InvariantCulture`.
- Metadata: `SELECT ttlValue, ttlUnit FROM tables() WHERE table_name = $1`. **QuestDB normalises the unit**: 7 days
shows `1 WEEK`, 14 days `2 WEEK`, 30 days `30 DAY`, none `0 HOUR`. The bootstrapper compares in hours
(`HOUR`=1, `DAY`=24, `WEEK`=168; any other unit counts as different) and issues `ALTER` only when they differ.
- `ALTER … SET TTL` is a WAL transaction: `tables()` shows the old value until it is applied (25–100 ms). Do not
re-read to verify without the Q4 wait.
- Partitions older than TTL are dropped on commit. With the default wall-clock mode a future-dated sample does **not**
expire current data.
- TTL cannot go on the `PARTITION BY YEAR` `hist_tag` table; it is never needed there.
## Q6 — timestamp precision
```
raw sent 2026-03-01T12:00:00.1234567(Utc) -> read 2026-03-01T12:00:00.1234560 micros=1772366400123456 exact=False
raw9 sent 2026-03-01T12:00:01.1234569(Utc) -> read 2026-03-01T12:00:01.1234560 micros=1772366401123456 exact=False
pre sent 2026-03-01T12:01:00.1234560(Utc) -> read 2026-03-01T12:01:00.1234560 micros=1772366460123456 exact=True
```
**Decision:** Npgsql **truncates** sub-µs ticks (…1234567 → …123456; not rounded). That is truncation **toward zero
relative to 2000-01-01**, so before 2000 it rounds *up* (Q13). A pre-truncated (`ticks - ticks % 10`) value
round-trips exactly. This confirms R6: always `Round` first.
## Q7 — concurrent `CREATE TABLE IF NOT EXISTS`
```
Q7 concurrent CREATE TABLE IF NOT EXISTS (4 connections x 20 rounds)
errors: none
plain CREATE on existing: SqlState=00000 MessageText='table already exists'
```
(An earlier run with 2 connections × 20 rounds also had no errors.) **Decision:** `IF NOT EXISTS` was race-free here.
The S11 catch rule stays as defence: catch a `PostgresException` whose `MessageText` contains **`already exists`**,
then re-check with `SELECT count() FROM tables() WHERE table_name = $1`, and rethrow when the table is not there. Never
key on SQLSTATE (it is always `00000`).
## Q8 — bucket statistics
Seed (3 × 10 s buckets from `start = t0 + 0.5 s`): bucket 0 holds Good 5, Uncertain 2, Bad 9, a usable String, Good 2
(a min tie with the Uncertain 2), a tombstone of 100, and Good 7 at the last µs of the bucket. Bucket 1 holds Good 1 at
the edge, an Empty `BadNoCommunication` and an Empty with Good status. Bucket 2 holds Bad 3 only. Another tag sits in
range.
```
Simple family (GROUP BY alias):
cols: b, total_count, usable_count, good_count, bad_count, non_numeric_count, avg_value, min_value, max_value, sum_value
0 | 6 | 5 | 4 | 1 | 1 | 4 | 2 | 7 | 16
1 | 3 | 1 | 1 | 2 | 0 | 1 | 1 | 1 | 1
2 | 1 | 0 | 0 | 1 | 0 | NULL | NULL | NULL | NULL
[OK] simple GROUP BY <expression> (no subquery) [OK] simple implicit group-by
StartEnd form A (first()/last()):
0 | 6 | 5 | 4 | 1 | 1 | 1 | 5 | NULL | NULL | 00:00:00.5000000 | 1 | 7 | NULL | NULL | 00:00:10.4999990
1 | 3 | 1 | 1 | 2 | 0 | 1 | 1 | NULL | NULL | 00:00:10.5000000 | 1 | 1 | NULL | NULL | 00:00:10.5000000
2 | 1 | 0 | 0 | 1 | 0 | 0 | NULL | NULL | NULL | NULL | 0 | NULL | NULL | NULL | NULL
StartEnd form B (row_number window): identical rows [OK] filter-in-subquery [OK] extra predicate in ON
ActualTime (min): 0 ... | 2 | 00:00:01.5000000 1 ... | 1 | 00:00:10.5000000 2 ... | NULL | NULL
ActualTime (max): 0 ... | 7 | 00:00:10.4999990 1 ... | 1 | 00:00:10.5000000 2 ... | NULL | NULL
[OK] arg_min(ts, num_value) exists? 0 | 00:00:01.5 | 00:00:10.499999 1 | 00:00:10.5 | 00:00:10.5
randomized cross-check (60k rows over 3 days (4 partitions), 2 tags, 7 min buckets, random statuses/kinds/tombstones):
buckets=617 rowsInWindow=28423 mismatches=0; first()/last() 23 ms, window 21 ms, actual-time min+max 26 ms;
arg_min/arg_max (one stmt) 11 ms, tie-break mismatches vs earliest-ts rule = 0
```
Every hand-computed value matches: bucket 0 averages (5+2+2+7)/4 = 4, the tombstone is invisible, the sample one µs
before the edge stays in bucket 0, the edge sample is in bucket 1, and both Empty rows count as bad whatever their
status. The randomized cross-check compares counts, first/last usable times, sums and min/max actual times against a
client-side computation. It used a µs-aligned start that is not on a second boundary, and integer-rounded values to
force ties.
**Decision (the confirmed forms, with the SQL Server family split kept):**
- Classified subquery `s`. Status is stored as the **unsigned** uint in a LONG, so usable = `status < 2147483648` and
good = `status < 1073741824`, both also needing `kind <> 0`:
```sql
SELECT (cast(ts AS LONG) * 10 - $1) / $2 AS b, ts, kind, num_value, int_value, str_value,
CASE WHEN status < 2147483648 AND kind <> 0 THEN 1 ELSE 0 END AS u,
CASE WHEN status < 1073741824 AND kind <> 0 THEN 1 ELSE 0 END AS g
FROM {p}sample WHERE tag = $3 AND ts >= $4 AND ts < $5 AND NOT deleted
```
`$1` = window start in **ticks since the Unix epoch** (`start.Ticks - DateTime.UnixEpoch.Ticks`, `Bigint`); `$2` =
interval in ticks (`Bigint`); `$4`/`$5` = rounded window bounds. `cast(ts AS LONG)` is µs since 1970, so `*10` is
ticks. That is exact because both sides are already µs-aligned (R6). At year 9999 `µs*10` ≈ 2.5e18 < 9.2e18, so it
cannot overflow.
- Common counts: `count() AS total_count, sum(u) AS usable_count, sum(g) AS good_count, count() - sum(u) AS bad_count,
sum(CASE WHEN u = 1 AND kind IN (4, 5) THEN 1 ELSE 0 END) AS non_numeric_count`.
- **Simple:** `SELECT b, <counts>, avg(CASE WHEN u = 1 THEN num_value END) AS avg_value, min(…), max(…), sum(…) FROM (<s>)
GROUP BY b ORDER BY b`.
- **StartEnd:** the window form, which is the same shape as SQL Server:
`WITH s AS (<s>), st AS (SELECT b, <counts> FROM s GROUP BY b), f AS (SELECT b, kind, num_value, int_value,
str_value, ts, row_number() OVER (PARTITION BY b ORDER BY ts) AS rf, row_number() OVER (PARTITION BY b ORDER BY ts
DESC) AS rl FROM s WHERE u = 1) SELECT st.*, x.kind, x.num_value, x.int_value, x.str_value, x.ts, y.kind, …, y.ts
FROM st LEFT JOIN (SELECT * FROM f WHERE rf = 1) x ON x.b = st.b LEFT JOIN (SELECT * FROM f WHERE rl = 1) y ON
y.b = st.b ORDER BY st.b`. A bucket without a usable sample has NULL `ts` but **kind `0`** (BYTE has no NULL), so the
evaluator must key "no first/last" on the NULL time.
- **ActualTime:** `… r AS (SELECT b, num_value, ts, row_number() OVER (PARTITION BY b ORDER BY num_value ASC|DESC, ts
ASC) AS rn FROM s WHERE u = 1 AND num_value IS NOT NULL) SELECT st.*, r.num_value, r.ts FROM st LEFT JOIN (SELECT *
FROM r WHERE rn = 1) r ON r.b = st.b ORDER BY st.b`.
- **One query per node holds for every family.** The S11 two-statement fallback is not needed. `first()/last()` and
`arg_min/arg_max` also gave correct answers, but their tie order is not specified by an `ORDER BY`. The window form
states the earliest-timestamp tie-break explicitly, so it is the one to use.
- `SAMPLE BY` is not used.
## Q9 — AtTime neighbour lookup
200k rows (2 tags × 100k, ~1.5 s apart, 5 % tombstones). 102 request times, including one before the first sample,
one exact hit and one after the last sample.
```
form A SQL (one time): SELECT * FROM (SELECT 0 AS req, 0 AS dir, ts, kind, num_value, int_value, str_value, status FROM spike_q9
WHERE tag = $1 AND ts <= $2 AND NOT deleted ORDER BY ts DESC LIMIT 1) UNION ALL SELECT * FROM (SELECT 0 AS req, 1 AS dir, ts, ...
WHERE tag = $1 AND ts > $2 AND NOT deleted ORDER BY ts ASC LIMIT 1)
form A chunk= 1: 61 ms for 102 times, rows=202, mismatches=0
form A chunk= 10: 20 ms ... chunk= 25: 18 ms chunk= 50: 19 ms chunk= 60: 18 ms chunk= 63: 16 ms (all mismatches=0)
form A chunk= 64: FAIL 'SQL model is too complex to evaluate' (128 UNION ALL branches)
form A chunk=102: FAIL 'SQL model is too complex to evaluate' (204 UNION ALL branches)
form B: 1 ms for 102 times (prev only), rows=102, mismatches=0 [ASOF JOIN of a long_sequence(1) UNION ALL request set]
```
**Decision: form A, chunked `LIMIT 1` seeks, at most 50 request times (100 `UNION ALL` branches) per statement.**
QuestDB rejects 128 or more branches. The literal `req` index and `dir` (0 = previous-or-equal, 1 = next) columns are
generated integers, not user input; `$1` is the tag and `$2..` the rounded times. It returns both neighbours in one
shape and costs ~0.2 ms per time. ASOF JOIN is faster, but it only gives the previous neighbour, so a second
seek form would still be needed for the next one. That does not justify two code paths at this scale.
## Q10 — long strings and SYMBOL capacity
```
[OK] 400-char tag (SYMBOL) + 400-char str_value (VARCHAR)
[OK] 5000-char str_value
30,000 distinct SYMBOL values inserted in 723 ms
count()=30002 | count_distinct(tag)=30002
400-char round trip: tag equal=True str equal=True lengths=400,400 (includes non-ASCII αβγ…)
tag | SYMBOL | symbolCached=True | symbolCapacity=16384
point lookup by tag among 30k symbols: count=1 0 ms
```
**Decision:** VARCHAR and SYMBOL take 400 characters (and VARCHAR takes 5,000). SYMBOL grows past its capacity hint
(16,384) without error, so 30k tags is fine. No `CAPACITY` clause is needed.
## Q11 — existing-timestamp lookup for a batch
```
[OK] range form: WHERE tag=$1 AND ts >= $2 AND ts <= $3 (tombstones included) -> 10 rows, incl. 00:00:08 deleted=True
[OK] IN-list form: ts IN ($2..$11) -> 4 rows (00, 06, 12, 18)
[OK] tombstone visible to range lookup -> 00:00:08 | True
```
**Decision:** both work with parameters. **Live writer (§2.2 step 2): the range form**
`SELECT ts FROM {p}sample WHERE tag = $1 AND ts >= $2 AND ts <= $3` (no `NOT deleted`, so tombstones count as present),
then a client-side set test. **Updater (sparse item times): the IN-list form** `SELECT ts, deleted FROM {p}sample WHERE
tag = $1 AND ts IN ($2, …)`, chunked at 1,000 times.
## Q12 — test cleanup and prefix listing
```
LIKE 't12_%' (note: _ is a wildcard) -> t12_sample, t12xsample <- WRONG: matches t12xsample
LIKE $1 with escaped _ ('t12\_%') -> t12_sample
starts_with(table_name, $1) -> t12_sample
left(table_name, 4) = $1 -> t12_sample
[OK] DROP TABLE IF EXISTS t12_sample (exists) [OK] DROP TABLE IF EXISTS t12_sample (already gone)
```
**Decision:** cleanup is `DROP TABLE IF EXISTS {name}`. List with **`SELECT table_name FROM tables() WHERE
starts_with(table_name, $1)`**. Never use `LIKE 'prefix%'`, because `_` in `hist_` is a wildcard.
## Q13 — binding extreme DateTimes
Seeds: 1970-01-01, 1990-05-05T00:00:00.123456, 2026-03-01, 2200-01-01. "Round" is R6's rule: `MinValue` → `+10 ticks`,
then `ticks - ticks % 10`.
```
DateTime.MinValue raw -> server sees µs -9222425352054775808 (Npgsql sends -infinity)
DateTime.MinValue Round -> 0001-01-01T00:00:00.0000010 (µs -62135596799999999)
DateTime.MaxValue raw -> server sees µs -9222425352054775809 (Npgsql sends +infinity)
DateTime.MaxValue Round -> 9999-12-31T23:59:59.9999990 (µs 253402300799999999)
0001-01-01 raw -> µs -9222425352054775808 (== MinValue -> -infinity)
1601-01-01 raw/Round -> 1601-01-01T00:00:00 (µs -11644473600000000)
9999-12-31T23:59:59.9999999 raw -> µs -9222425352054775809 (== MaxValue -> +infinity)
1969-12-31T23:59:59.9999995 raw -> 1970-01-01T00:00:00.0000000 (µs 0) Round -> 1969-12-31T23:59:59.9999990 (µs -1)
1990-05-05T00:00:00.1234567 raw -> 1990-05-05T00:00:00.1234570 Round -> ...1234560
1990-05-05T00:00:00.1234565 raw -> 1990-05-05T00:00:00.1234570 Round -> ...1234560
2026-03-01T00:00:00.0000005 raw -> 2026-03-01T00:00:00.0000000 Round -> same
range queries WHERE ts >= $1 AND ts < $2:
Min..Max raw count=0 <- silently empty
Min..Max Round count=4
0001..9999.9999999 Round count=4
1601..t0 Round count=2
1990.1234565..1990.1234575 raw count=0 <- seed .123456 missed (bound arrived as .123457)
1990.1234565..1990.1234575 Round count=1
inserting extreme designated timestamps:
[FAIL] insert ts=1969-12-31T23:59:59: PostgresException: 00000: designated timestamp before 1970-01-01 is not allowed
[FAIL] insert ts=1601-01-01: ... [FAIL] insert ts=Min+10 ticks: ...
[OK] insert ts=9999-12-31 Round -> 9999-12-31T23:59:59.9999990
```
**Decision:** R6's `BindTimestamp` is **confirmed necessary and sufficient**.
- A raw `DateTime.MinValue`/`MaxValue` (and anything equal to them) becomes ±infinity, and QuestDB stores that as a
near-`long.MinValue` value. A `[Min, Max)` range then matches **nothing**, with no error.
- A raw pre-2000 sub-µs value rounds **up**, which shifts a bound across a stored sample.
- With Round, every bound behaves.
- **New constraint:** a designated timestamp **before 1970-01-01** cannot be inserted. Reads with pre-1970 bounds work.
See the spec §2.2 ruling candidate.
## DDL and DeleteRaw paging (extra)
```
[OK] CREATE TABLE IF NOT EXISTS spike_hist_sample (...) TIMESTAMP(ts) PARTITION BY DAY TTL 30 DAYS WAL DEDUP UPSERT KEYS(ts, tag)
[OK] CREATE TABLE IF NOT EXISTS spike_hist_event (...) TIMESTAMP(ts) PARTITION BY DAY TTL 30 DAYS WAL DEDUP UPSERT KEYS(ts, event_id)
[OK] CREATE TABLE IF NOT EXISTS spike_hist_tag (...) TIMESTAMP(ts) PARTITION BY YEAR WAL DEDUP UPSERT KEYS(ts, name)
[OK] re-run (idempotent) x3
spike_hist_event | DAY | True | True | 30 | DAY
spike_hist_sample | DAY | True | True | 30 | DAY
spike_hist_tag | YEAR | True | True | 0 | HOUR
hist_tag upsert-by-name at ts=1970-01-01 (tagA twice, tagB once):
1970-01-01T00:00:00 | tagA | 4 | degF | second
1970-01-01T00:00:00 | tagB | 1 | NULL | NULL
DeleteRaw paging (page = 1000 rows): 2500 live rows + 1 outside range
pages=3 tombstones=2500 in 103 ms
deleted=False 1 deleted=True 2500
```
DeleteRaw loop, as confirmed: `SELECT ts FROM {p}sample WHERE tag = $1 AND ts >= $2 AND ts < $3 AND NOT deleted ORDER BY
ts LIMIT 1000` → one 1,000-row tombstone INSERT → the Q4 wait → continue from `last ts + 10 ticks`. The wait matters:
without it the next page can re-read rows it just tombstoned.
## Rulings candidates raised by the spike
1. **Pre-1970 timestamps cannot be stored** (Q13). QuestDB rejects a designated timestamp before 1970-01-01, and the
whole multi-row INSERT fails. Proposed:
- the live writer drops such samples and logs one Warning per batch. Retrying cannot succeed, and sending the
sample would fail the tag's whole group into #538 quarantine;
- HistoryUpdate Insert/Replace/Update items before 1970 return `BadOutOfRange`;
- DeleteAtTime/DeleteRaw find nothing there and return `BadNoEntryExists`;
- reads are unaffected.
Conformance tests that seed pre-1970 data are skipped for QuestDB.
2. **`long.MinValue` is QuestDB's LONG NULL** (sentinel check). An Int64 sample of exactly `long.MinValue` would read
back `int_value` NULL, and `FromColumns` would then serve **0**. Proposed: the QuestDB read path repairs it. When
`kind = Int64`, `int_value IS NULL` and `num_value = -9.223372036854775808E18` (exact in a double), serve
`long.MinValue`. That is a local fix-up before the shared `FromColumns`, with no write-side change.
3. **TTL metadata is unit-normalised and applied asynchronously** (Q5). The bootstrapper compares in hours and does not
re-read straight after an `ALTER`.