diff --git a/docker-dev/docker-compose.questdb.yml b/docker-dev/docker-compose.questdb.yml new file mode 100644 index 00000000..a86422c9 --- /dev/null +++ b/docker-dev/docker-compose.questdb.yml @@ -0,0 +1,28 @@ +# docker-dev/docker-compose.questdb.yml — optional QuestDB fixture for the QuestDB historian backend (#666). +# +# Up (touches ONLY the questdb service; it has no depends_on, and --no-deps makes that explicit): +# docker compose -f docker-dev/docker-compose.yml -f docker-dev/docker-compose.questdb.yml up -d --no-deps questdb +# +# Image pinned by the T0 spike (docs/plans/2026-10-10-questdb-spike.md): questdb/questdb:10.0.1, +# pushed to Docker Hub 2026-08-24 (≥ 2 weeks before 2026-10-10; ≥ 8.2.3, the first release with table TTL). +# The image ships /usr/bin/curl, so the healthcheck uses it against the min-HTTP health endpoint on :9003. +# +# PG wire -> localhost:18812 (Host=localhost;Port=18812;Username=admin;Password=quest;Database=qdb;...) +# Console -> http://localhost:19000 +services: + questdb: + image: questdb/questdb:10.0.1 + container_name: otopcua-dev-questdb-1 + ports: + - "18812:8812" # PG wire + - "19000:9000" # web console / REST + volumes: + - otopcua-questdb-data:/var/lib/questdb + healthcheck: + test: ["CMD-SHELL", "curl -fs http://localhost:9003/status || exit 1"] + interval: 10s + timeout: 5s + retries: 12 +volumes: + otopcua-questdb-data: + name: otopcua-questdb-data diff --git a/docs/plans/2026-10-10-questdb-historian-design.md b/docs/plans/2026-10-10-questdb-historian-design.md index 17574b8c..63ec36ed 100644 --- a/docs/plans/2026-10-10-questdb-historian-design.md +++ b/docs/plans/2026-10-10-questdb-historian-design.md @@ -81,28 +81,64 @@ Host changes: QuestDB has no schemas, so the tables carry a configurable `TablePrefix` (default `hist_`, `[A-Za-z0-9_]+`). All three are **WAL tables with `DEDUP UPSERT KEYS`**, so re-sending a row with the same key replaces it. +**T0-confirmed** against `questdb/questdb:10.0.1` (`docs/plans/2026-10-10-questdb-spike.md` §DDL, Q3, Q5, Q10). The +`TTL n DAYS` clause goes **between `PARTITION BY` and `WAL`**. Placed after the dedup clause it fails with `unexpected +token [TTL]`. It is emitted only when `RetentionDays` is set, as an inlined integer, because it cannot be a parameter. + ```sql CREATE TABLE IF NOT EXISTS hist_sample ( ts TIMESTAMP, tag SYMBOL, kind BYTE, num_value DOUBLE, int_value LONG, str_value VARCHAR, status LONG, deleted BOOLEAN, server_ts TIMESTAMP -) TIMESTAMP(ts) PARTITION BY DAY WAL DEDUP UPSERT KEYS(ts, tag) [TTL n DAYS]; +) TIMESTAMP(ts) PARTITION BY DAY [TTL n DAYS] WAL DEDUP UPSERT KEYS(ts, tag); CREATE TABLE IF NOT EXISTS hist_event ( - ts TIMESTAMP, event_id VARCHAR, source SYMBOL, - -- plus the remaining hist.Event columns one-for-one (docs/historian/sqlserver/001_initial.sql), QuestDB-typed -) TIMESTAMP(ts) PARTITION BY DAY WAL DEDUP UPSERT KEYS(ts, event_id) [TTL n DAYS]; + ts TIMESTAMP, event_id VARCHAR, source SYMBOL, alarm_id VARCHAR, alarm_name VARCHAR, alarm_type_name VARCHAR, + severity INT, event_kind VARCHAR, message VARCHAR, user_name VARCHAR, comment VARCHAR, received_utc TIMESTAMP +) TIMESTAMP(ts) PARTITION BY DAY [TTL n DAYS] WAL DEDUP UPSERT KEYS(ts, event_id); CREATE TABLE IF NOT EXISTS hist_tag ( ts TIMESTAMP, name SYMBOL, data_type BYTE, eng_unit VARCHAR, description VARCHAR, updated_utc TIMESTAMP ) TIMESTAMP(ts) PARTITION BY YEAR WAL DEDUP UPSERT KEYS(ts, name); -- ts is always 1970-01-01, which makes this an upsert by name ``` -Column details: -- `status` holds the uint status code as a LONG. -- `int` also holds Booleans as 0/1, matching SQL Server. -- `kind` is `HistorianValueKind`. -- The exact column types and the TTL syntax are confirmed by T0 against the pinned image. T0 amends this section - with what it finds. +`hist_event` maps `hist.Event` one-for-one: +- `EventTimeUtc` → `ts`, `EventId` → `event_id`, `SourceName` → `source`; +- `AlarmId`/`AlarmName`/`AlarmTypeName` → `alarm_id`/`alarm_name`/`alarm_type_name`; +- `Severity` → `severity`, `EventKind` → `event_kind`, `Message` → `message`; +- `UserName` → `user_name`, `Comment` → `comment`, `ReceivedTimeUtc` → `received_utc`. + +`event_id VARCHAR` works as a dedup key (T0 Q3), so re-sending an event replaces it. The S4 fallback (no dedup on +`hist_event`) is not used. + +Column details (**T0-confirmed**): +- `status` holds the uint status code **unsigned** in a LONG (0..4294967295, never negative). So "severity Bad" is + `status >= 2147483648`, not `< 0` as in SQL Server's signed `int`. +- `int_value` also holds Booleans as 0/1, matching SQL Server. +- `kind` is `HistorianValueKind` in a `BYTE`. +- Read-back CLR types: + + | Column type | Npgsql type name | CLR type / reader call | + |---|---|---| + | `BYTE` | `smallint` | `Int16` (`GetInt16`) | + | `SYMBOL`, `VARCHAR` | `character varying` | `String` | + | `LONG` | `bigint` | `Int64` | + | `INT` | `integer` | `Int32` | + | `BOOLEAN` | `boolean` | `Boolean` | + | `DOUBLE` | `double precision` | `Double` | + | `TIMESTAMP` | `timestamp without time zone` | `DateTime`, **`Kind=Unspecified`**; the reader applies `SpecifyKind(…, Utc)` | + +- **NULL semantics:** + - DOUBLE NaN and ±Inf read back as NULL. + - `long.MinValue` (LONG) and `int.MinValue` (INT) are QuestDB's NULL sentinels. + - `BYTE` and `BOOLEAN` have **no NULL**: unset or DBNull reads `0` / `false`. So `deleted` defaults to `false`, and a + LEFT-JOINed "no row" yields `kind = 0`. Absence must be detected on a NULL `ts`, not on `kind`. +- **Ruling candidate (T0):** an Int64 sample of exactly `long.MinValue` stores `int_value` as NULL. The QuestDB read + path repairs it before the shared `FromColumns`: `kind = Int64 ∧ int_value IS NULL ∧ num_value = -2^63` ⇒ + `long.MinValue`. That double is exact. Without the repair the sample would be served as 0. +- A designated timestamp **before 1970-01-01 cannot be inserted** (`designated timestamp before 1970-01-01 is not + allowed`). See §2.2. +- Tag names: SYMBOL and VARCHAR take 400 characters (non-ASCII included). SYMBOL grows past its capacity hint + (16,384) with no error, and was tested with 30,000 distinct tags. No `CAPACITY` clause is needed. ### 2.2 Live writes (recorder) — insert only if absent, the same as SQL Server @@ -110,16 +146,45 @@ Column details: to microseconds before this step, so microsecond collisions also collapse. 2. For each tag, fetch the timestamps that already exist in the batch's `[min ts, max ts]` range. **Tombstones are included**, so a re-sent outbox batch can never resurrect a deleted row or overwrite a HistoryUpdate change. -3. Insert only the new rows, in one parameterised multi-row `INSERT` per tag group. + T0-confirmed form (Q11): `SELECT ts FROM {p}sample WHERE tag = $1 AND ts >= $2 AND ts <= $3`, with no `NOT + deleted`, followed by a client-side set test. +3. Insert only the new rows with parameterised multi-row `INSERT … VALUES (…),(…)`, **at most 1,000 rows per + statement** (T0-confirmed Q2). QuestDB reads the Bind parameter count as a signed int16, so the hard limit is + 32,767 parameters (3,640 nine-column rows). Within one statement a duplicate key keeps the **last** row, which is + why step 1's "first wins" is done client-side. 4. Stored-value rules are shared with SQL Server: - NaN/±Inf, or any Empty with a non-Bad status, is stored Empty with `BadOutOfRange` or `BadNoData`. - Comms-loss outage markers are stored as given. - A tag name over 400 characters is a per-tag failure. + - **Ruling candidate (T0 Q13):** a sample timestamped before 1970-01-01 cannot be stored, because QuestDB rejects + the whole INSERT. The writer drops such samples before the INSERT and logs one Warning per batch. A retry could + never succeed, and sending the sample would push the tag's whole group into the #538 quarantine. 5. Any SQL or transport failure makes the write return `false`, so the recorder retries and the #538 per-tag quarantine still applies. It never throws. The recorder is Primary-only (#537), so step 2 does not race another live writer. The recorder waits for WAL -visibility, bounded by `VisibilityTimeout` (plan R5). +visibility, bounded by `VisibilityTimeout` (plan R5), using the §2.3 wait. + +Binding (T0-confirmed Q1/Q6/Q13): +- Every timestamp parameter is `NpgsqlDbType.TimestampTz` with `DateTimeKind.Utc`, after `Round`. `Timestamp` + `Utc` + throws in Npgsql. Npgsql truncates sub-µs ticks **toward 2000-01-01**, which rounds *up* before 2000. A raw + `DateTime.MinValue`/`MaxValue` is sent as ±infinity, and a `[Min, Max)` range then silently matches nothing. R6's + `BindTimestamp` (round; `MinValue` → `+10` ticks; `MaxValue` → `−9` ticks) fixes all three. +- The other parameter types: + + | Column | `NpgsqlDbType` | + |---|---| + | `tag`/`str_value`/other strings | `Varchar` | + | `kind`/`data_type` | `Smallint` | + | `num_value` | `Double` | + | `int_value`/`status` | `Bigint` | + | `deleted` | `Boolean` | + | `severity` | `Integer` | + + NULL is `DBNull.Value` with the column's type. +- Every QuestDB server error carries **SQLSTATE `00000`**. `QuestDbConnection`'s classification therefore treats a + `PostgresException` as a server-side rejection, judged by `MessageText`. Any other `NpgsqlException`, `IOException`, + `SocketException` or timeout is transport (transient). ### 2.3 HistoryUpdate (soft delete) @@ -140,6 +205,25 @@ Further rules: - **Read-your-writes.** After its inserts, the updater polls `wal_tables()` until the table's applied transaction reaches the sequencer transaction it produced, up to `VisibilityTimeout`. On a timeout the write is kept, the codes are reported as normal, and a Warning is logged. +- **The visibility wait (`QuestDbWalVisibility`, T0-confirmed Q4)** is used by the writer and the updater (R5): + 1. After the INSERT(s) return: `SELECT sequencerTxn FROM wal_tables() WHERE name = $1` (`$1` = the full table name, + `Varchar`) → `target`. Each autocommitted INSERT is exactly one sequencer transaction. + 2. Poll `SELECT writerTxn, suspended FROM wal_tables() WHERE name = $1` about every 10 ms until `writerTxn >= + target`. Stop early, with a Warning, if `suspended` is true. Stop at `VisibilityTimeout`. + - Measured: median 0 ms, p95 6 ms, max 13 ms. Up to 7/20 inserts were not visible to an immediate re-read, so the + wait is required. A concurrent writer can only raise `target`. The key-poll fallback (S11) is not used. +- Existence checks for sparse item times (Insert/Replace/Update/DeleteAtTime) use `SELECT ts, deleted FROM {p}sample + WHERE tag = $1 AND ts IN ($2, …)`, chunked at 1,000 times (T0 Q11). +- **DeleteRaw (T0-confirmed), page size 1,000:** + 1. `SELECT ts FROM {p}sample WHERE tag = $1 AND ts >= $2 AND ts < $3 AND NOT deleted ORDER BY ts LIMIT 1000`. + 2. Write one 1,000-row tombstone INSERT. + 3. Run the visibility wait. + 4. Continue from `last ts + 10 ticks`, until a page comes back empty. + The wait between pages is required, or the next page re-reads rows just tombstoned. Measured: 2,500 rows, 3 pages, + 103 ms. +- **Ruling candidate (T0 Q13):** an Insert, Replace or Update item timestamped before 1970-01-01 returns + `BadOutOfRange`, because QuestDB cannot store it. Deletes there find nothing and return `BadNoEntryExists`. Pre-1970 + conformance seeds are skipped for QuestDB. - The existence checks for Insert and Replace read committed data. If a live recorder sample at the *same* timestamp is still in flight in the WAL, it can race the updater. Updates normally target past timestamps, so this is accepted and documented. @@ -150,20 +234,56 @@ When `RetentionDays` is set, the bootstrapper applies TTL to `hist_sample` and ` the configured value changes. QuestDB then drops expired day partitions, tombstones included. When unset, data is kept forever. `hist_tag` never expires. +T0-confirmed (Q5): +- **Create:** `… PARTITION BY DAY TTL n DAYS WAL …` (§2.1). +- **Change:** `ALTER TABLE {p}sample SET TTL n DAYS`. **Remove** (`RetentionDays` unset while the table has a TTL): + `ALTER TABLE … SET TTL 0 DAYS`. +- `n` cannot be a `$` parameter (`invalid syntax, should be but was $1`). It is the validated integer, + formatted with `InvariantCulture`. +- TTL is not allowed on the `PARTITION BY YEAR` `hist_tag` table (`TTL value must be an integer multiple of the + partition size`). +- **Reading the current TTL:** `SELECT ttlValue, ttlUnit FROM tables() WHERE table_name = $1`. QuestDB **normalises + the unit**: 7 days reads `1 WEEK`, 14 days `2 WEEK`, 30 days `30 DAY`, none `0 HOUR`. The bootstrapper compares in + hours (`HOUR`=1, `DAY`=24, `WEEK`=168; any other unit counts as different) and issues an `ALTER` only on a + difference. +- `ALTER … SET TTL` is applied through the WAL, so `tables()` shows the old value for 25–100 ms. The bootstrapper does + not re-read to verify. +- Expiry is checked against the **wall clock**: the image default is `cairo.ttl.use.wall.clock=true`. A future-dated + sample does not expire current data (T0 Q5c). +- **Table listing:** `SELECT table_name FROM tables() WHERE starts_with(table_name, $1)`. Never `LIKE 'hist_%'`, + because `_` is a wildcard. +- **Concurrent bootstrap:** `CREATE TABLE IF NOT EXISTS` from 4 connections × 20 rounds produced no errors. The S11 + defence stays: catch a `PostgresException` whose `MessageText` contains `already exists`, re-check `tables()`, and + rethrow if the table is absent. Never key on SQLSTATE (always `00000`). + ## 3. Reads Every query filters `NOT deleted`. Every value goes through `StoredSample`, so the quality clamp is the same as SQL Server's. - **Raw.** A range on `ts`, `ORDER BY ts [DESC]`, `LIMIT @top` (`maxValuesPerNode == 0` ⇒ unbounded), at most `top` samples, `ContinuationPoint: null`; the node manager pages by re-reading from the last timestamp (plan R8). The dedup key forbids same-instant ties. Bounds follow the existing data-source contract. -- **AtTime.** Find the previous-or-equal neighbour and the next neighbour of each requested time. The default is - `LIMIT 1` seeks on `ts`, batched into chunked statements; T0 picks `ASOF JOIN` for the previous neighbour if it - measures better. `AtTimeEvaluator` then decides the result. +- **AtTime (T0-confirmed Q9).** Find the previous-or-equal neighbour and the next neighbour of each requested time + with `LIMIT 1` seeks, `UNION ALL`-ed, **at most 50 request times (100 branches) per statement**. QuestDB rejects + 128 or more branches with `SQL model is too complex to evaluate`. Per time `k`, with `$1` = tag and `$j` = the + rounded time: + ```sql + SELECT * FROM (SELECT k AS req, 0 AS dir, ts, kind, num_value, int_value, str_value, status FROM {p}sample + WHERE tag = $1 AND ts <= $j AND NOT deleted ORDER BY ts DESC LIMIT 1) + UNION ALL + SELECT * FROM (SELECT k AS req, 1 AS dir, ts, kind, num_value, int_value, str_value, status FROM {p}sample + WHERE tag = $1 AND ts > $j AND NOT deleted ORDER BY ts ASC LIMIT 1) + ``` + `k` and `dir` are generated integer literals. Measured: 102 times in about 16–20 ms over 100k rows per tag, with no + mismatches. `ASOF JOIN` was faster (1 ms), but it yields only the previous neighbour, so it was not chosen. + `AtTimeEvaluator` then decides the result. - **Events.** Filtered by source and `ts` range, ordered ascending (reversed windows are swapped, as built today), and limited to `maxEvents`. -- **Processed.** **One query per node.** - - Bucket index: `(cast(ts AS LONG) - @startMicros) / @intervalMicros`, using microsecond values from `BucketPlan`, - grouped by the index. +- **Processed.** **One query per node** (T0-confirmed for every family; the S11 two-statement fallback is not needed). + - Bucket index (T0-confirmed): `(cast(ts AS LONG) * 10 - $1) / $2`. + - `cast(ts AS LONG)` is µs since 1970. + - `$1` = the rounded window start in **ticks since the Unix epoch** (`start.Ticks − DateTime.UnixEpoch.Ticks`). + - `$2` = the interval in ticks. Both are `NpgsqlDbType.Bigint`. + - The result is exact because both sides are µs-aligned (R6), and it cannot overflow before year 9999. - The query returns the full `BucketRow` statistics. Usable, good, bad and uncertain counts come from `sum(CASE …)` over the status column. - It does **not** use `SAMPLE BY`, whose alignment rules differ from ours. Computing the index ourselves keeps the @@ -171,13 +291,43 @@ Server's. 100k cap. - The time of the min or max value needs an arg-min style lookup. It is computed with a window function, or with a second statement if QuestDB lacks one; T0 decides. `AggregateEvaluator` then produces the values. + - **Confirmed forms (T0 Q8).** They mirror `SqlAggregateQueries`'s three families and column order. They were checked + against hand-computed seeds and a 60k-row randomized cross-check across 4 partitions: 617 buckets, 0 mismatches. + - Classified subquery `s`: + ```sql + SELECT (cast(ts AS LONG) * 10 - $1) / $2 AS b, ts, kind, num_value, int_value, str_value, + CASE WHEN status < 2147483648 AND kind <> 0 THEN 1 ELSE 0 END AS u, + CASE WHEN status < 1073741824 AND kind <> 0 THEN 1 ELSE 0 END AS g + FROM {p}sample WHERE tag = $3 AND ts >= $4 AND ts < $5 AND NOT deleted + ``` + - Counts (ordinals 1–5 after `b`): `count() AS total_count, sum(u) AS usable_count, sum(g) AS good_count, count() - + sum(u) AS bad_count, sum(CASE WHEN u = 1 AND kind IN (4, 5) THEN 1 ELSE 0 END) AS non_numeric_count`. + - **Simple:** `SELECT b, , avg(CASE WHEN u = 1 THEN num_value END) AS avg_value, min(CASE WHEN u = 1 THEN + num_value END) AS min_value, max(…) AS max_value, sum(…) AS sum_value FROM () GROUP BY b ORDER BY b`. + - **StartEnd:** + ```sql + WITH s AS (), st AS (SELECT b, FROM s GROUP BY b), + f AS (SELECT b, kind, num_value, int_value, str_value, ts, + row_number() OVER (PARTITION BY b ORDER BY ts) AS rf, + row_number() OVER (PARTITION BY b ORDER BY ts DESC) AS rl FROM s WHERE u = 1) + SELECT st.*, x.kind, x.num_value, x.int_value, x.str_value, x.ts, y.kind, y.num_value, y.int_value, y.str_value, y.ts + FROM st LEFT JOIN (SELECT * FROM f WHERE rf = 1) x ON x.b = st.b + LEFT JOIN (SELECT * FROM f WHERE rl = 1) y ON y.b = st.b ORDER BY st.b + ``` + - **ActualTime:** the same `s`/`st`, then `r AS (SELECT b, num_value, ts, row_number() OVER (PARTITION BY b ORDER + BY num_value ASC|DESC, ts ASC) AS rn FROM s WHERE u = 1 AND num_value IS NOT NULL) SELECT st.*, r.num_value, r.ts + FROM st LEFT JOIN (SELECT * FROM r WHERE rn = 1) r ON r.b = st.b ORDER BY st.b`. + - A bucket with no usable sample returns NULL times but **`kind = 0`** (BYTE has no NULL). The mapper keys absence on + the NULL `ts`. + - `first()/last()` and `arg_min/arg_max` also returned correct values, but no `ORDER BY` fixes their tie order. The + window forms state the earliest-timestamp tie-break explicitly and are the ones to use. - **Errors.** A failed read fails only that node, under the same data-source contract as SQL Server. ## 4. Configuration (`ServerHistorian:QuestDb`) | Key | Default | Validation | |---|---|---| -| `ConnectionString` | — | Required when `Backend=QuestDb` and a consumer is enabled. An Npgsql string, e.g. `Host=questdb;Port=8812;Username=admin;Password=quest;Database=qdb;ServerCompatibilityMode=NoTypeLoading` (T0 confirms the exact keywords). | +| `ConnectionString` | — | Required when `Backend=QuestDb` and a consumer is enabled. An Npgsql string. **T0-confirmed:** `Host=questdb;Port=8812;Username=admin;Password=quest;Database=qdb;ServerCompatibilityMode=NoTypeLoading` (the docker-dev overlay from the host: `Host=localhost;Port=18812;…`). `ServerCompatibilityMode=NoTypeLoading` is kept: it skips the `pg_type` load, and every confirmed form was run with it. The string also works without it. `No Reset On Close` is **not** needed, because pooled open → close → reopen works. | | `TablePrefix` | `hist_` | `^[A-Za-z0-9_]+$`. Only checked when a consumer is enabled. | | `RetentionDays` | unset | ≥ 1 when set | | `CommandTimeout` | `00:00:30` | > 0 | diff --git a/docs/plans/2026-10-10-questdb-spike.md b/docs/plans/2026-10-10-questdb-spike.md new file mode 100644 index 00000000..510dc1cc --- /dev/null +++ b/docs/plans/2026-10-10-questdb-spike.md @@ -0,0 +1,411 @@ +# QuestDB historian backend — T0 feasibility spike + +**Status:** run 2026-10-11 against a live container. Gitea #666. Spec: `docs/plans/2026-10-10-questdb-historian-design.md` +(§2–§4 amended in place from this doc; every confirmed form there is marked **T0-confirmed**). + +## Fixture + +| Item | Value | +|---|---| +| Image | **`questdb/questdb:10.0.1`** — Docker Hub `tag_last_pushed` **2026-08-24T14:00:08Z** (48 days before 2026-10-10) | +| Why that tag | Newest exact `x.y.z` tag ≥ 14 days older than 2026-10-10. Newer tags are `nightly*` / `patch-*` only. Floor for TTL is **8.2.3**; 10.0.1 is well above it. | +| Build | `SELECT build()` → `Build Information: QuestDB 10.0.1, JDK 25.0.2, Commit Hash 7a391566ac827e0d8c269b91f2415c16b0a32a84` | +| Overlay | `docker-dev/docker-compose.questdb.yml` — container `otopcua-dev-questdb-1`, ports `18812:8812` (PG wire) and `19000:9000` (console/REST), named volume `otopcua-questdb-data` (explicit `name:` so it is not project-prefixed) | +| Up | `docker compose -f docker-dev/docker-compose.yml -f docker-dev/docker-compose.questdb.yml up -d --no-deps questdb` (a `--dry-run` showed only the volume + `otopcua-dev-questdb-1` being created; nothing else in the `otopcua-dev` project is touched) | +| Healthcheck | the image ships `/usr/bin/curl` (`command -v curl` → `/usr/bin/curl`; no `wget`), so `curl -fs http://localhost:9003/status` is kept. Container reached `healthy`. | +| Harness | throwaway net10.0 console app, Npgsql **10.0.3** (the repo's central pin, `Directory.Packages.props` line 97), in the session scratchpad `questdb-spike/` with its own `Directory.Build.props`, `Directory.Packages.props` and `nuget.config`. Nothing was added to the repo. | + +### Connection string (confirmed) + +``` +Host=localhost;Port=18812;Username=admin;Password=quest;Database=qdb;ServerCompatibilityMode=NoTypeLoading +``` + +Inside the docker network the same string uses `Host=questdb;Port=8812`. + +## Q1 — keywords, parameter types, CLR types + +``` + [OK] open+SELECT 1 with 'Host=localhost;Port=18812;Username=admin;Password=quest;Database=qdb' + [OK] open+SELECT 1 with '...;ServerCompatibilityMode=NoTypeLoading' + [OK] open->close->reopen x5 on one data source (no 'No Reset On Close') + [OK] reopen after a parameterised INSERT (prepared-ish state) + [FAIL] ts Utc + Timestamp: ArgumentException: Cannot write DateTime with Kind=UTC to PostgreSQL type 'timestamp without time zone', consider using 'timestamp with time zone'. ... + [OK] ts Utc + TimestampTz + [OK] ts Utc + inferred + [OK] ts Unspecified + Timestamp + [FAIL] ts Local + TimestampTz: ArgumentException: Cannot write DateTime with Kind=Local to PostgreSQL type 'timestamp with time zone', only UTC is supported. ... + [OK] kind bound as Integer (int4) + [OK] tag bound as Text + [OK] NaN double insert + [OK] +Inf double insert + cols: ts:timestamp without time zone/DateTime, tag:character varying/String, kind:smallint/Int16, num_value:double precision/Double, + int_value:bigint/Int64, str_value:character varying/String, status:bigint/Int64, deleted:boolean/Boolean, server_ts:timestamp without time zone/DateTime + 2026-01-02T03:04:05.1234560(Unspecified) | ts Utc + TimestampTz | 3 | 1.5 | 7 | s | 2147483648 | False | 2026-10-11T04:56:27.1103710(Unspecified) + ... + 2026-01-02T03:04:11.1234560(Unspecified) | nan | 0 | NULL | NULL | NULL | NULL | False | NULL + 2026-01-02T03:04:12.1234560(Unspecified) | inf | 0 | NULL | NULL | NULL | NULL | False | NULL + integer arithmetic with $n: + cols: ts::LONG - $1:bigint/Int64 [OK] cast(ts AS LONG) - $1 (Bigint) + cols: (ts::LONG * 10 - $1) / $2:bigint/Int64 [OK] (Bigint) [OK] (Integer) +``` + +Sentinel NULLs (extra check): + +``` + cols: ts, l:bigint/Int64, i:integer/Int32, b:smallint/Int16, z:boolean/Boolean, l = NULL:boolean + 2026-03-01T00:00:00 | NULL | NULL | 0 | False | True <- l = long.MinValue, i = int.MinValue written + 2026-03-01T00:00:01 | -9223372036854775807 | -2147483647 | 0 | False | False + 2026-03-01T00:00:02 | NULL | NULL | 0 | False | True <- BYTE/BOOLEAN bound as DBNull +``` + +**Decisions** +- Both connection strings work. **Keep `ServerCompatibilityMode=NoTypeLoading`**: it skips the `pg_type` load, and every + form in this doc was run with it on. **`No Reset On Close` is not needed**: open → close → reopen on one pooled + `NpgsqlDataSource` works. +- **Timestamps are bound as `NpgsqlDbType.TimestampTz` with `DateTimeKind.Utc`.** `Timestamp` + `Utc` throws in Npgsql + before reaching the server. `Unspecified` + `Timestamp` also works, but one rule is enough. +- **Timestamps read back as `DateTime` with `Kind=Unspecified`**, wall clock = UTC. The reader must + `DateTime.SpecifyKind(r.GetDateTime(i), DateTimeKind.Utc)`. +- CLR types returned: `BYTE` → `smallint` / **`Int16`** (`GetInt16`); `SYMBOL` → `character varying` / `String`; + `VARCHAR` → `String`; `LONG` → `Int64`; `INT` → `Int32`; `BOOLEAN` → `Boolean`; `TIMESTAMP` → `DateTime`; + `DOUBLE` → `Double`. `count()`/`sum(int)` → `Int64`. +- Parameter types: `BYTE` ← `Smallint` (`Integer` also accepted); `SYMBOL`/`VARCHAR` ← `Varchar` (`Text` also accepted); + `LONG` ← `Bigint`; `BOOLEAN` ← `Boolean`; `DOUBLE` ← `Double`; a NULL is `DBNull.Value` with the column's + `NpgsqlDbType`. +- `$n` in integer arithmetic: **bind as `NpgsqlDbType.Bigint`** (Integer also worked for small values; Bigint is the + rule because tick offsets exceed int32). +- **DOUBLE NaN and ±Inf read back as NULL.** **`LONG` `long.MinValue` and `INT` `int.MinValue` are QuestDB's NULL + sentinels** and read back NULL. **`BYTE` and `BOOLEAN` have no NULL**: unset or DBNull reads `0` / `false`. So + `deleted` defaults to `false` and `kind` of a never-written field is `0` (Empty). +- **Every server error carries `SqlState = 00000`.** Classification cannot use SQLSTATE: a `PostgresException` is a + server-side rejection (decide by `MessageText`), any other `NpgsqlException`/`IOException`/`SocketException`/timeout + is transport. + +## Q2 — multi-row parameterised INSERT and batch size + +``` + [OK] 1000 rows x 9 params = 9000 params 88 ms + [OK] 2000 rows x 9 params = 18000 params 167 ms + [OK] 3640 rows x 9 params = 32760 params 342 ms + [FAIL] 3641 rows x 9 params = 32769 params: PostgresException: 00000: invalid parameter count [parameterCount=-32767, offset=225670 + [FAIL] 5000 rows x 9 params = 45000 params: PostgresException: 00000: invalid parameter count [parameterCount=-20536, ... + [FAIL] 7282 rows x 9 params = 65538 params: NpgsqlException: A statement cannot have more than 65535 parameters + 20 x 1000-row statements: fails=0, 122 ms total + count()=27251 | count_distinct(ts)=27251 expected rows = 27251 +``` + +**Decision:** QuestDB reads the Bind parameter count as a **signed int16**, so the real limit is **32,767 parameters per +statement**, not Npgsql's 65,535. **Batch size = 1,000 rows per `INSERT … VALUES (…),(…)`** (9,000 parameters for +`hist_sample`), which is reliable and far under the limit. **DeleteRaw page size = 1,000 rows** (see the DDL section). + +## Q3 — `DEDUP UPSERT KEYS` + +``` + before: 2026-03-01T00:00:00 | a | 1 | False 2026-03-01T00:00:00 | b | 2 | False + after re-insert of (t0,'a') as tombstone: + 2026-03-01T00:00:00 | a | NULL | True 2026-03-01T00:00:00 | b | 2 | False + NOT deleted: 2026-03-01T00:00:00 | b | 2 + same key twice in one statement (10 then 20): 2026-03-01T00:00:01 | c | 20 + hist_event with event_id VARCHAR as a dedup key: + [OK] CREATE hist_event-shape with DEDUP UPSERT KEYS(ts, event_id) on VARCHAR + [OK] insert e1 'first', e2, then e1 'second' + 2026-03-01T00:00:00 | e1 | src1 | 500 | second | NULL + 2026-03-01T00:00:00 | e2 | src1 | 500 | other | NULL +``` + +**Decision:** re-inserting a key replaces the whole row; a tombstone written at `(ts, tag)` hides the row from `NOT +deleted`. **Within one statement the LAST duplicate wins**, so the writer's "first value wins" rule must be applied +client-side before the INSERT (spec §2.2 step 1 already does this). **`event_id VARCHAR` works as a dedup key**, so the +S4 fallback (no dedup on `hist_event`) is **not needed**. + +## Q4 — `wal_tables()` and the visibility wait + +``` + cols: name:varchar, suspended:boolean, writerTxn:bigint, bufferedTxnSize:bigint, sequencerTxn:bigint, + errorTag:varchar, errorMessage:varchar, memoryPressure:integer + run 0: seqTxn before=0 after=1 visibleImmediately=False waitMs=13 visibleAfterWait=True + run 1: seqTxn before=1 after=2 visibleImmediately=False waitMs=0 visibleAfterWait=True + ... + run 16: seqTxn before=16 after=17 visibleImmediately=False waitMs=6 visibleAfterWait=True + lag ms: min=0 median=0 p95=6 max=13; visible-without-wait 17/20 +``` + +(An earlier run: visible without waiting 13/20, max 6 ms.) Every autocommitted INSERT is exactly one sequencer +transaction. + +**Decision (the form T3's `QuestDbWalVisibility` implements):** +1. After the INSERT(s) return, on the same or any connection: + `SELECT sequencerTxn FROM wal_tables() WHERE name = $1` (`$1` = the table name, `Varchar`) → `target`. +2. Poll `SELECT writerTxn, suspended FROM wal_tables() WHERE name = $1` every ~10 ms until `writerTxn >= target`, or + `suspended` is true, or `VisibilityTimeout` elapses. + +A concurrent writer can only make `target` larger, which lengthens the wait slightly and never shortens it. The wait is +needed: up to 7/20 inserts were not yet visible on the next query. The S11 key-poll fallback is **not needed**. + +## Q5 — TTL + +``` + [FAIL] CREATE ... PARTITION BY DAY WAL DEDUP UPSERT KEYS(ts, tag) TTL 7 DAYS (TTL last): unexpected token [TTL] + [OK] CREATE ... PARTITION BY DAY TTL 7 DAYS WAL DEDUP UPSERT KEYS(ts, tag) (TTL before WAL) + spike_q5 | DAY | True | True | ttlValue=1 | ttlUnit=WEEK + [FAIL] ALTER TABLE spike_q5 SET TTL $1 DAYS (parameter): invalid syntax, should be but was $1 + [FAIL] TTL 7 DAYS on PARTITION BY YEAR table: TTL value must be an integer multiple of the partition size (its time interval) +Q5b ALTER ... SET TTL visibility in tables() + SET TTL 30 DAYS : seqTxn 6->7; applied after 100 ms; tables() now ttlValue=30 ttlUnit=DAY + SET TTL 14 DAYS : seqTxn 7->8; applied after 94 ms; tables() now ttlValue=2 ttlUnit=WEEK + SET TTL 1 DAY : seqTxn 8->9; applied after 59 ms; tables() now ttlValue=1 ttlUnit=DAY + SET TTL 0 DAYS : seqTxn 9->10; applied after 25 ms; tables() now ttlValue=0 ttlUnit=HOUR + SET TTL 3 DAYS : seqTxn 10->11; applied after 60 ms; tables() now ttlValue=3 ttlUnit=DAY + after inserting now-20d and now into a TTL 7 DAYS table: + new | 2026-10-11T05:03:03 +Q5c TTL reference clock (future-dated row): + after inserting a now+30d row, then one more ordinary commit: + d-3 | 2026-10-08 now | 2026-10-11 now+1h | 2026-10-11 d+30 | 2026-11-10 +``` + +Image `conf/server.conf`: `#cairo.ttl.use.wall.clock=true` — "when true (default), TTL enforcement uses wall clock +time to prevent accidental data loss when future timestamps are inserted". + +**Decision:** +- Create syntax: **`… TIMESTAMP(ts) PARTITION BY DAY TTL n DAYS WAL DEDUP UPSERT KEYS(…)`** — TTL goes **between + `PARTITION BY` and `WAL`**. Alter: **`ALTER TABLE t SET TTL n DAYS`**; remove: **`ALTER TABLE t SET TTL 0 DAYS`**. + `n` cannot be a parameter; it is the validated integer `RetentionDays`, formatted with `InvariantCulture`. +- Metadata: `SELECT ttlValue, ttlUnit FROM tables() WHERE table_name = $1`. **QuestDB normalises the unit**: 7 days + shows `1 WEEK`, 14 days `2 WEEK`, 30 days `30 DAY`, none `0 HOUR`. The bootstrapper compares in hours + (`HOUR`=1, `DAY`=24, `WEEK`=168; any other unit counts as different) and issues `ALTER` only when they differ. +- `ALTER … SET TTL` is a WAL transaction: `tables()` shows the old value until it is applied (25–100 ms). Do not + re-read to verify without the Q4 wait. +- Partitions older than TTL are dropped on commit. With the default wall-clock mode a future-dated sample does **not** + expire current data. +- TTL cannot go on the `PARTITION BY YEAR` `hist_tag` table; it is never needed there. + +## Q6 — timestamp precision + +``` + raw sent 2026-03-01T12:00:00.1234567(Utc) -> read 2026-03-01T12:00:00.1234560 micros=1772366400123456 exact=False + raw9 sent 2026-03-01T12:00:01.1234569(Utc) -> read 2026-03-01T12:00:01.1234560 micros=1772366401123456 exact=False + pre sent 2026-03-01T12:01:00.1234560(Utc) -> read 2026-03-01T12:01:00.1234560 micros=1772366460123456 exact=True +``` + +**Decision:** Npgsql **truncates** sub-µs ticks (…1234567 → …123456; not rounded). That is truncation **toward zero +relative to 2000-01-01**, so before 2000 it rounds *up* (Q13). A pre-truncated (`ticks - ticks % 10`) value +round-trips exactly. This confirms R6: always `Round` first. + +## Q7 — concurrent `CREATE TABLE IF NOT EXISTS` + +``` +Q7 concurrent CREATE TABLE IF NOT EXISTS (4 connections x 20 rounds) + errors: none + plain CREATE on existing: SqlState=00000 MessageText='table already exists' +``` + +(An earlier run with 2 connections × 20 rounds also had no errors.) **Decision:** `IF NOT EXISTS` was race-free here. +The S11 catch rule stays as defence: catch a `PostgresException` whose `MessageText` contains **`already exists`**, +then re-check with `SELECT count() FROM tables() WHERE table_name = $1`, and rethrow when the table is not there. Never +key on SQLSTATE (it is always `00000`). + +## Q8 — bucket statistics + +Seed (3 × 10 s buckets from `start = t0 + 0.5 s`): bucket 0 holds Good 5, Uncertain 2, Bad 9, a usable String, Good 2 +(a min tie with the Uncertain 2), a tombstone of 100, and Good 7 at the last µs of the bucket. Bucket 1 holds Good 1 at +the edge, an Empty `BadNoCommunication` and an Empty with Good status. Bucket 2 holds Bad 3 only. Another tag sits in +range. + +``` + Simple family (GROUP BY alias): + cols: b, total_count, usable_count, good_count, bad_count, non_numeric_count, avg_value, min_value, max_value, sum_value + 0 | 6 | 5 | 4 | 1 | 1 | 4 | 2 | 7 | 16 + 1 | 3 | 1 | 1 | 2 | 0 | 1 | 1 | 1 | 1 + 2 | 1 | 0 | 0 | 1 | 0 | NULL | NULL | NULL | NULL + [OK] simple GROUP BY (no subquery) [OK] simple implicit group-by + StartEnd form A (first()/last()): + 0 | 6 | 5 | 4 | 1 | 1 | 1 | 5 | NULL | NULL | 00:00:00.5000000 | 1 | 7 | NULL | NULL | 00:00:10.4999990 + 1 | 3 | 1 | 1 | 2 | 0 | 1 | 1 | NULL | NULL | 00:00:10.5000000 | 1 | 1 | NULL | NULL | 00:00:10.5000000 + 2 | 1 | 0 | 0 | 1 | 0 | 0 | NULL | NULL | NULL | NULL | 0 | NULL | NULL | NULL | NULL + StartEnd form B (row_number window): identical rows [OK] filter-in-subquery [OK] extra predicate in ON + ActualTime (min): 0 ... | 2 | 00:00:01.5000000 1 ... | 1 | 00:00:10.5000000 2 ... | NULL | NULL + ActualTime (max): 0 ... | 7 | 00:00:10.4999990 1 ... | 1 | 00:00:10.5000000 2 ... | NULL | NULL + [OK] arg_min(ts, num_value) exists? 0 | 00:00:01.5 | 00:00:10.499999 1 | 00:00:10.5 | 00:00:10.5 + randomized cross-check (60k rows over 3 days (4 partitions), 2 tags, 7 min buckets, random statuses/kinds/tombstones): + buckets=617 rowsInWindow=28423 mismatches=0; first()/last() 23 ms, window 21 ms, actual-time min+max 26 ms; + arg_min/arg_max (one stmt) 11 ms, tie-break mismatches vs earliest-ts rule = 0 +``` + +Every hand-computed value matches: bucket 0 averages (5+2+2+7)/4 = 4, the tombstone is invisible, the sample one µs +before the edge stays in bucket 0, the edge sample is in bucket 1, and both Empty rows count as bad whatever their +status. The randomized cross-check compares counts, first/last usable times, sums and min/max actual times against a +client-side computation. It used a µs-aligned start that is not on a second boundary, and integer-rounded values to +force ties. + +**Decision (the confirmed forms, with the SQL Server family split kept):** +- Classified subquery `s`. Status is stored as the **unsigned** uint in a LONG, so usable = `status < 2147483648` and + good = `status < 1073741824`, both also needing `kind <> 0`: + ```sql + SELECT (cast(ts AS LONG) * 10 - $1) / $2 AS b, ts, kind, num_value, int_value, str_value, + CASE WHEN status < 2147483648 AND kind <> 0 THEN 1 ELSE 0 END AS u, + CASE WHEN status < 1073741824 AND kind <> 0 THEN 1 ELSE 0 END AS g + FROM {p}sample WHERE tag = $3 AND ts >= $4 AND ts < $5 AND NOT deleted + ``` + `$1` = window start in **ticks since the Unix epoch** (`start.Ticks - DateTime.UnixEpoch.Ticks`, `Bigint`); `$2` = + interval in ticks (`Bigint`); `$4`/`$5` = rounded window bounds. `cast(ts AS LONG)` is µs since 1970, so `*10` is + ticks. That is exact because both sides are already µs-aligned (R6). At year 9999 `µs*10` ≈ 2.5e18 < 9.2e18, so it + cannot overflow. +- Common counts: `count() AS total_count, sum(u) AS usable_count, sum(g) AS good_count, count() - sum(u) AS bad_count, + sum(CASE WHEN u = 1 AND kind IN (4, 5) THEN 1 ELSE 0 END) AS non_numeric_count`. +- **Simple:** `SELECT b, , avg(CASE WHEN u = 1 THEN num_value END) AS avg_value, min(…), max(…), sum(…) FROM () + GROUP BY b ORDER BY b`. +- **StartEnd:** the window form, which is the same shape as SQL Server: + `WITH s AS (), st AS (SELECT b, FROM s GROUP BY b), f AS (SELECT b, kind, num_value, int_value, + str_value, ts, row_number() OVER (PARTITION BY b ORDER BY ts) AS rf, row_number() OVER (PARTITION BY b ORDER BY ts + DESC) AS rl FROM s WHERE u = 1) SELECT st.*, x.kind, x.num_value, x.int_value, x.str_value, x.ts, y.kind, …, y.ts + FROM st LEFT JOIN (SELECT * FROM f WHERE rf = 1) x ON x.b = st.b LEFT JOIN (SELECT * FROM f WHERE rl = 1) y ON + y.b = st.b ORDER BY st.b`. A bucket without a usable sample has NULL `ts` but **kind `0`** (BYTE has no NULL), so the + evaluator must key "no first/last" on the NULL time. +- **ActualTime:** `… r AS (SELECT b, num_value, ts, row_number() OVER (PARTITION BY b ORDER BY num_value ASC|DESC, ts + ASC) AS rn FROM s WHERE u = 1 AND num_value IS NOT NULL) SELECT st.*, r.num_value, r.ts FROM st LEFT JOIN (SELECT * + FROM r WHERE rn = 1) r ON r.b = st.b ORDER BY st.b`. +- **One query per node holds for every family.** The S11 two-statement fallback is not needed. `first()/last()` and + `arg_min/arg_max` also gave correct answers, but their tie order is not specified by an `ORDER BY`. The window form + states the earliest-timestamp tie-break explicitly, so it is the one to use. +- `SAMPLE BY` is not used. + +## Q9 — AtTime neighbour lookup + +200k rows (2 tags × 100k, ~1.5 s apart, 5 % tombstones). 102 request times, including one before the first sample, +one exact hit and one after the last sample. + +``` + form A SQL (one time): SELECT * FROM (SELECT 0 AS req, 0 AS dir, ts, kind, num_value, int_value, str_value, status FROM spike_q9 + WHERE tag = $1 AND ts <= $2 AND NOT deleted ORDER BY ts DESC LIMIT 1) UNION ALL SELECT * FROM (SELECT 0 AS req, 1 AS dir, ts, ... + WHERE tag = $1 AND ts > $2 AND NOT deleted ORDER BY ts ASC LIMIT 1) + form A chunk= 1: 61 ms for 102 times, rows=202, mismatches=0 + form A chunk= 10: 20 ms ... chunk= 25: 18 ms chunk= 50: 19 ms chunk= 60: 18 ms chunk= 63: 16 ms (all mismatches=0) + form A chunk= 64: FAIL 'SQL model is too complex to evaluate' (128 UNION ALL branches) + form A chunk=102: FAIL 'SQL model is too complex to evaluate' (204 UNION ALL branches) + form B: 1 ms for 102 times (prev only), rows=102, mismatches=0 [ASOF JOIN of a long_sequence(1) UNION ALL request set] +``` + +**Decision: form A, chunked `LIMIT 1` seeks, at most 50 request times (100 `UNION ALL` branches) per statement.** +QuestDB rejects 128 or more branches. The literal `req` index and `dir` (0 = previous-or-equal, 1 = next) columns are +generated integers, not user input; `$1` is the tag and `$2..` the rounded times. It returns both neighbours in one +shape and costs ~0.2 ms per time. ASOF JOIN is faster, but it only gives the previous neighbour, so a second +seek form would still be needed for the next one. That does not justify two code paths at this scale. + +## Q10 — long strings and SYMBOL capacity + +``` + [OK] 400-char tag (SYMBOL) + 400-char str_value (VARCHAR) + [OK] 5000-char str_value + 30,000 distinct SYMBOL values inserted in 723 ms + count()=30002 | count_distinct(tag)=30002 + 400-char round trip: tag equal=True str equal=True lengths=400,400 (includes non-ASCII αβγ…) + tag | SYMBOL | symbolCached=True | symbolCapacity=16384 + point lookup by tag among 30k symbols: count=1 0 ms +``` + +**Decision:** VARCHAR and SYMBOL take 400 characters (and VARCHAR takes 5,000). SYMBOL grows past its capacity hint +(16,384) without error, so 30k tags is fine. No `CAPACITY` clause is needed. + +## Q11 — existing-timestamp lookup for a batch + +``` + [OK] range form: WHERE tag=$1 AND ts >= $2 AND ts <= $3 (tombstones included) -> 10 rows, incl. 00:00:08 deleted=True + [OK] IN-list form: ts IN ($2..$11) -> 4 rows (00, 06, 12, 18) + [OK] tombstone visible to range lookup -> 00:00:08 | True +``` + +**Decision:** both work with parameters. **Live writer (§2.2 step 2): the range form** +`SELECT ts FROM {p}sample WHERE tag = $1 AND ts >= $2 AND ts <= $3` (no `NOT deleted`, so tombstones count as present), +then a client-side set test. **Updater (sparse item times): the IN-list form** `SELECT ts, deleted FROM {p}sample WHERE +tag = $1 AND ts IN ($2, …)`, chunked at 1,000 times. + +## Q12 — test cleanup and prefix listing + +``` + LIKE 't12_%' (note: _ is a wildcard) -> t12_sample, t12xsample <- WRONG: matches t12xsample + LIKE $1 with escaped _ ('t12\_%') -> t12_sample + starts_with(table_name, $1) -> t12_sample + left(table_name, 4) = $1 -> t12_sample + [OK] DROP TABLE IF EXISTS t12_sample (exists) [OK] DROP TABLE IF EXISTS t12_sample (already gone) +``` + +**Decision:** cleanup is `DROP TABLE IF EXISTS {name}`. List with **`SELECT table_name FROM tables() WHERE +starts_with(table_name, $1)`**. Never use `LIKE 'prefix%'`, because `_` in `hist_` is a wildcard. + +## Q13 — binding extreme DateTimes + +Seeds: 1970-01-01, 1990-05-05T00:00:00.123456, 2026-03-01, 2200-01-01. "Round" is R6's rule: `MinValue` → `+10 ticks`, +then `ticks - ticks % 10`. + +``` + DateTime.MinValue raw -> server sees µs -9222425352054775808 (Npgsql sends -infinity) + DateTime.MinValue Round -> 0001-01-01T00:00:00.0000010 (µs -62135596799999999) + DateTime.MaxValue raw -> server sees µs -9222425352054775809 (Npgsql sends +infinity) + DateTime.MaxValue Round -> 9999-12-31T23:59:59.9999990 (µs 253402300799999999) + 0001-01-01 raw -> µs -9222425352054775808 (== MinValue -> -infinity) + 1601-01-01 raw/Round -> 1601-01-01T00:00:00 (µs -11644473600000000) + 9999-12-31T23:59:59.9999999 raw -> µs -9222425352054775809 (== MaxValue -> +infinity) + 1969-12-31T23:59:59.9999995 raw -> 1970-01-01T00:00:00.0000000 (µs 0) Round -> 1969-12-31T23:59:59.9999990 (µs -1) + 1990-05-05T00:00:00.1234567 raw -> 1990-05-05T00:00:00.1234570 Round -> ...1234560 + 1990-05-05T00:00:00.1234565 raw -> 1990-05-05T00:00:00.1234570 Round -> ...1234560 + 2026-03-01T00:00:00.0000005 raw -> 2026-03-01T00:00:00.0000000 Round -> same + range queries WHERE ts >= $1 AND ts < $2: + Min..Max raw count=0 <- silently empty + Min..Max Round count=4 + 0001..9999.9999999 Round count=4 + 1601..t0 Round count=2 + 1990.1234565..1990.1234575 raw count=0 <- seed .123456 missed (bound arrived as .123457) + 1990.1234565..1990.1234575 Round count=1 + inserting extreme designated timestamps: + [FAIL] insert ts=1969-12-31T23:59:59: PostgresException: 00000: designated timestamp before 1970-01-01 is not allowed + [FAIL] insert ts=1601-01-01: ... [FAIL] insert ts=Min+10 ticks: ... + [OK] insert ts=9999-12-31 Round -> 9999-12-31T23:59:59.9999990 +``` + +**Decision:** R6's `BindTimestamp` is **confirmed necessary and sufficient**. +- A raw `DateTime.MinValue`/`MaxValue` (and anything equal to them) becomes ±infinity, and QuestDB stores that as a + near-`long.MinValue` value. A `[Min, Max)` range then matches **nothing**, with no error. +- A raw pre-2000 sub-µs value rounds **up**, which shifts a bound across a stored sample. +- With Round, every bound behaves. +- **New constraint:** a designated timestamp **before 1970-01-01** cannot be inserted. Reads with pre-1970 bounds work. + See the spec §2.2 ruling candidate. + +## DDL and DeleteRaw paging (extra) + +``` + [OK] CREATE TABLE IF NOT EXISTS spike_hist_sample (...) TIMESTAMP(ts) PARTITION BY DAY TTL 30 DAYS WAL DEDUP UPSERT KEYS(ts, tag) + [OK] CREATE TABLE IF NOT EXISTS spike_hist_event (...) TIMESTAMP(ts) PARTITION BY DAY TTL 30 DAYS WAL DEDUP UPSERT KEYS(ts, event_id) + [OK] CREATE TABLE IF NOT EXISTS spike_hist_tag (...) TIMESTAMP(ts) PARTITION BY YEAR WAL DEDUP UPSERT KEYS(ts, name) + [OK] re-run (idempotent) x3 + spike_hist_event | DAY | True | True | 30 | DAY + spike_hist_sample | DAY | True | True | 30 | DAY + spike_hist_tag | YEAR | True | True | 0 | HOUR + hist_tag upsert-by-name at ts=1970-01-01 (tagA twice, tagB once): + 1970-01-01T00:00:00 | tagA | 4 | degF | second + 1970-01-01T00:00:00 | tagB | 1 | NULL | NULL + DeleteRaw paging (page = 1000 rows): 2500 live rows + 1 outside range + pages=3 tombstones=2500 in 103 ms + deleted=False 1 deleted=True 2500 +``` + +DeleteRaw loop, as confirmed: `SELECT ts FROM {p}sample WHERE tag = $1 AND ts >= $2 AND ts < $3 AND NOT deleted ORDER BY +ts LIMIT 1000` → one 1,000-row tombstone INSERT → the Q4 wait → continue from `last ts + 10 ticks`. The wait matters: +without it the next page can re-read rows it just tombstoned. + +## Rulings candidates raised by the spike + +1. **Pre-1970 timestamps cannot be stored** (Q13). QuestDB rejects a designated timestamp before 1970-01-01, and the + whole multi-row INSERT fails. Proposed: + - the live writer drops such samples and logs one Warning per batch. Retrying cannot succeed, and sending the + sample would fail the tag's whole group into #538 quarantine; + - HistoryUpdate Insert/Replace/Update items before 1970 return `BadOutOfRange`; + - DeleteAtTime/DeleteRaw find nothing there and return `BadNoEntryExists`; + - reads are unaffected. + Conformance tests that seed pre-1970 data are skipped for QuestDB. +2. **`long.MinValue` is QuestDB's LONG NULL** (sentinel check). An Int64 sample of exactly `long.MinValue` would read + back `int_value` NULL, and `FromColumns` would then serve **0**. Proposed: the QuestDB read path repairs it. When + `kind = Int64`, `int_value IS NULL` and `num_value = -9.223372036854775808E18` (exact in a double), serve + `long.MinValue`. That is a local fix-up before the shared `FromColumns`, with no write-side change. +3. **TTL metadata is unit-normalised and applied asynchronously** (Q5). The bootstrapper compares in hours and does not + re-read straight after an `ALTER`.