Joseph Doherty
452 lines
29 KiB
Markdown
452 lines
29 KiB
Markdown
# QuestDB historian backend — T0 feasibility spike
|
||
|
||
**Status:** run 2026-10-11 against a live container. Gitea #666. Spec: `docs/plans/2026-10-10-questdb-historian-design.md`
|
||
(§2–§4 amended in place from this doc; every confirmed form there is marked **T0-confirmed**).
|
||
|
||
## Fixture
|
||
|
||
| Item | Value |
|
||
|---|---|
|
||
| Image | **`questdb/questdb:10.0.1`** — Docker Hub `tag_last_pushed` **2026-08-24T14:00:08Z** (48 days before 2026-10-10) |
|
||
| Why that tag | Newest exact `x.y.z` tag ≥ 14 days older than 2026-10-10. Newer tags are `nightly*` / `patch-*` only. Floor for TTL is **8.2.3**; 10.0.1 is well above it. |
|
||
| Build | `SELECT build()` → `Build Information: QuestDB 10.0.1, JDK 25.0.2, Commit Hash 7a391566ac827e0d8c269b91f2415c16b0a32a84` |
|
||
| Overlay | `docker-dev/docker-compose.questdb.yml` — container `otopcua-dev-questdb-1`, ports `18812:8812` (PG wire) and `19000:9000` (console/REST), named volume `otopcua-questdb-data` (explicit `name:` so it is not project-prefixed) |
|
||
| Up | `docker compose -f docker-dev/docker-compose.yml -f docker-dev/docker-compose.questdb.yml up -d --no-deps questdb` (a `--dry-run` showed only the volume + `otopcua-dev-questdb-1` being created; nothing else in the `otopcua-dev` project is touched) |
|
||
| Healthcheck | the image ships `/usr/bin/curl` (`command -v curl` → `/usr/bin/curl`; no `wget`), so `curl -fs http://localhost:9003/status` is kept. Container reached `healthy`. |
|
||
| Harness | throwaway net10.0 console app, Npgsql **10.0.3** (the repo's central pin, `Directory.Packages.props` line 97), in the session scratchpad `questdb-spike/` with its own `Directory.Build.props`, `Directory.Packages.props` and `nuget.config`. Nothing was added to the repo. |
|
||
|
||
### Connection string (confirmed)
|
||
|
||
```
|
||
Host=localhost;Port=18812;Username=admin;Password=quest;Database=qdb;ServerCompatibilityMode=NoTypeLoading
|
||
```
|
||
|
||
Inside the docker network the same string uses `Host=questdb;Port=8812`.
|
||
|
||
## Q1 — keywords, parameter types, CLR types
|
||
|
||
```
|
||
[OK] open+SELECT 1 with 'Host=localhost;Port=18812;Username=admin;Password=quest;Database=qdb'
|
||
[OK] open+SELECT 1 with '...;ServerCompatibilityMode=NoTypeLoading'
|
||
[OK] open->close->reopen x5 on one data source (no 'No Reset On Close')
|
||
[OK] reopen after a parameterised INSERT (prepared-ish state)
|
||
[FAIL] ts Utc + Timestamp: ArgumentException: Cannot write DateTime with Kind=UTC to PostgreSQL type 'timestamp without time zone', consider using 'timestamp with time zone'. ...
|
||
[OK] ts Utc + TimestampTz
|
||
[OK] ts Utc + inferred
|
||
[OK] ts Unspecified + Timestamp
|
||
[FAIL] ts Local + TimestampTz: ArgumentException: Cannot write DateTime with Kind=Local to PostgreSQL type 'timestamp with time zone', only UTC is supported. ...
|
||
[OK] kind bound as Integer (int4)
|
||
[OK] tag bound as Text
|
||
[OK] NaN double insert
|
||
[OK] +Inf double insert
|
||
cols: ts:timestamp without time zone/DateTime, tag:character varying/String, kind:smallint/Int16, num_value:double precision/Double,
|
||
int_value:bigint/Int64, str_value:character varying/String, status:bigint/Int64, deleted:boolean/Boolean, server_ts:timestamp without time zone/DateTime
|
||
2026-01-02T03:04:05.1234560(Unspecified) | ts Utc + TimestampTz | 3<Int16> | 1.5 | 7<Int64> | s | 2147483648<Int64> | False | 2026-10-11T04:56:27.1103710(Unspecified)
|
||
...
|
||
2026-01-02T03:04:11.1234560(Unspecified) | nan | 0<Int16> | NULL | NULL | NULL | NULL | False | NULL
|
||
2026-01-02T03:04:12.1234560(Unspecified) | inf | 0<Int16> | NULL | NULL | NULL | NULL | False | NULL
|
||
integer arithmetic with $n:
|
||
cols: ts::LONG - $1:bigint/Int64 [OK] cast(ts AS LONG) - $1 (Bigint)
|
||
cols: (ts::LONG * 10 - $1) / $2:bigint/Int64 [OK] (Bigint) [OK] (Integer)
|
||
```
|
||
|
||
Sentinel NULLs (extra check):
|
||
|
||
```
|
||
cols: ts, l:bigint/Int64, i:integer/Int32, b:smallint/Int16, z:boolean/Boolean, l = NULL:boolean
|
||
2026-03-01T00:00:00 | NULL | NULL | 0 | False | True <- l = long.MinValue, i = int.MinValue written
|
||
2026-03-01T00:00:01 | -9223372036854775807 | -2147483647 | 0 | False | False
|
||
2026-03-01T00:00:02 | NULL | NULL | 0 | False | True <- BYTE/BOOLEAN bound as DBNull
|
||
```
|
||
|
||
**Decisions**
|
||
- Both connection strings work. **Keep `ServerCompatibilityMode=NoTypeLoading`**: it skips the `pg_type` load, and every
|
||
form in this doc was run with it on. **`No Reset On Close` is not needed**: open → close → reopen on one pooled
|
||
`NpgsqlDataSource` works.
|
||
- **Timestamps are bound as `NpgsqlDbType.TimestampTz` with `DateTimeKind.Utc`.** `Timestamp` + `Utc` throws in Npgsql
|
||
before reaching the server. `Unspecified` + `Timestamp` also works, but one rule is enough.
|
||
- **Timestamps read back as `DateTime` with `Kind=Unspecified`**, wall clock = UTC. The reader must
|
||
`DateTime.SpecifyKind(r.GetDateTime(i), DateTimeKind.Utc)`.
|
||
- CLR types returned: `BYTE` → `smallint` / **`Int16`** (`GetInt16`); `SYMBOL` → `character varying` / `String`;
|
||
`VARCHAR` → `String`; `LONG` → `Int64`; `INT` → `Int32`; `BOOLEAN` → `Boolean`; `TIMESTAMP` → `DateTime`;
|
||
`DOUBLE` → `Double`. `count()`/`sum(int)` → `Int64`.
|
||
- Parameter types: `BYTE` ← `Smallint` (`Integer` also accepted); `SYMBOL`/`VARCHAR` ← `Varchar` (`Text` also accepted);
|
||
`LONG` ← `Bigint`; `BOOLEAN` ← `Boolean`; `DOUBLE` ← `Double`; a NULL is `DBNull.Value` with the column's
|
||
`NpgsqlDbType`.
|
||
- `$n` in integer arithmetic: **bind as `NpgsqlDbType.Bigint`** (Integer also worked for small values; Bigint is the
|
||
rule because tick offsets exceed int32).
|
||
- **DOUBLE NaN and ±Inf read back as NULL.** **`LONG` `long.MinValue` and `INT` `int.MinValue` are QuestDB's NULL
|
||
sentinels** and read back NULL. **`BYTE` and `BOOLEAN` have no NULL**: unset or DBNull reads `0` / `false`. So
|
||
`deleted` defaults to `false` and `kind` of a never-written field is `0` (Empty).
|
||
- **Every server error carries `SqlState = 00000`.** Classification cannot use SQLSTATE: a `PostgresException` is a
|
||
server-side rejection (decide by `MessageText`), any other `NpgsqlException`/`IOException`/`SocketException`/timeout
|
||
is transport.
|
||
|
||
## Q2 — multi-row parameterised INSERT and batch size
|
||
|
||
```
|
||
[OK] 1000 rows x 9 params = 9000 params 88 ms
|
||
[OK] 2000 rows x 9 params = 18000 params 167 ms
|
||
[OK] 3640 rows x 9 params = 32760 params 342 ms
|
||
[FAIL] 3641 rows x 9 params = 32769 params: PostgresException: 00000: invalid parameter count [parameterCount=-32767, offset=225670
|
||
[FAIL] 5000 rows x 9 params = 45000 params: PostgresException: 00000: invalid parameter count [parameterCount=-20536, ...
|
||
[FAIL] 7282 rows x 9 params = 65538 params: NpgsqlException: A statement cannot have more than 65535 parameters
|
||
20 x 1000-row statements: fails=0, 122 ms total
|
||
count()=27251 | count_distinct(ts)=27251 expected rows = 27251
|
||
```
|
||
|
||
**Decision:** QuestDB reads the Bind parameter count as a **signed int16**, so the real limit is **32,767 parameters per
|
||
statement**, not Npgsql's 65,535. **Batch size = 1,000 rows per `INSERT … VALUES (…),(…)`** (9,000 parameters for
|
||
`hist_sample`), which is reliable and far under the limit. **DeleteRaw page size = 1,000 rows** (see the DDL section).
|
||
|
||
## Q3 — `DEDUP UPSERT KEYS`
|
||
|
||
```
|
||
before: 2026-03-01T00:00:00 | a | 1 | False 2026-03-01T00:00:00 | b | 2 | False
|
||
after re-insert of (t0,'a') as tombstone:
|
||
2026-03-01T00:00:00 | a | NULL | True 2026-03-01T00:00:00 | b | 2 | False
|
||
NOT deleted: 2026-03-01T00:00:00 | b | 2
|
||
same key twice in one statement (10 then 20): 2026-03-01T00:00:01 | c | 20
|
||
hist_event with event_id VARCHAR as a dedup key:
|
||
[OK] CREATE hist_event-shape with DEDUP UPSERT KEYS(ts, event_id) on VARCHAR
|
||
[OK] insert e1 'first', e2, then e1 'second'
|
||
2026-03-01T00:00:00 | e1 | src1 | 500 | second | NULL
|
||
2026-03-01T00:00:00 | e2 | src1 | 500 | other | NULL
|
||
```
|
||
|
||
**Decision:** re-inserting a key replaces the whole row; a tombstone written at `(ts, tag)` hides the row from `NOT
|
||
deleted`. **Within one statement the LAST duplicate wins**, so the writer's "first value wins" rule must be applied
|
||
client-side before the INSERT (spec §2.2 step 1 already does this). **`event_id VARCHAR` works as a dedup key**, so the
|
||
S4 fallback (no dedup on `hist_event`) is **not needed**.
|
||
|
||
## Q4 — `wal_tables()` and the visibility wait
|
||
|
||
```
|
||
cols: name:varchar, suspended:boolean, writerTxn:bigint, bufferedTxnSize:bigint, sequencerTxn:bigint,
|
||
errorTag:varchar, errorMessage:varchar, memoryPressure:integer
|
||
run 0: seqTxn before=0 after=1 visibleImmediately=False waitMs=13 visibleAfterWait=True
|
||
run 1: seqTxn before=1 after=2 visibleImmediately=False waitMs=0 visibleAfterWait=True
|
||
...
|
||
run 16: seqTxn before=16 after=17 visibleImmediately=False waitMs=6 visibleAfterWait=True
|
||
lag ms: min=0 median=0 p95=6 max=13; visible-without-wait 17/20
|
||
```
|
||
|
||
(An earlier run: visible without waiting 13/20, max 6 ms.) Every autocommitted INSERT is exactly one sequencer
|
||
transaction.
|
||
|
||
**Decision (the form T3's `QuestDbWalVisibility` implements):**
|
||
1. After the INSERT(s) return, on the same or any connection:
|
||
`SELECT sequencerTxn FROM wal_tables() WHERE name = $1` (`$1` = the table name, `Varchar`) → `target`.
|
||
2. Poll `SELECT writerTxn, suspended FROM wal_tables() WHERE name = $1` every ~10 ms until `writerTxn >= target`, or
|
||
`suspended` is true, or `VisibilityTimeout` elapses.
|
||
|
||
A concurrent writer can only make `target` larger, which lengthens the wait slightly and never shortens it. The wait is
|
||
needed: up to 7/20 inserts were not yet visible on the next query. The S11 key-poll fallback is **not needed**.
|
||
|
||
## Q5 — TTL
|
||
|
||
```
|
||
[FAIL] CREATE ... PARTITION BY DAY WAL DEDUP UPSERT KEYS(ts, tag) TTL 7 DAYS (TTL last): unexpected token [TTL]
|
||
[OK] CREATE ... PARTITION BY DAY TTL 7 DAYS WAL DEDUP UPSERT KEYS(ts, tag) (TTL before WAL)
|
||
spike_q5 | DAY | True | True | ttlValue=1 | ttlUnit=WEEK
|
||
[FAIL] ALTER TABLE spike_q5 SET TTL $1 DAYS (parameter): invalid syntax, should be <number> <unit> but was $1
|
||
[FAIL] TTL 7 DAYS on PARTITION BY YEAR table: TTL value must be an integer multiple of the partition size (its time interval)
|
||
Q5b ALTER ... SET TTL visibility in tables()
|
||
SET TTL 30 DAYS : seqTxn 6->7; applied after 100 ms; tables() now ttlValue=30 ttlUnit=DAY
|
||
SET TTL 14 DAYS : seqTxn 7->8; applied after 94 ms; tables() now ttlValue=2 ttlUnit=WEEK
|
||
SET TTL 1 DAY : seqTxn 8->9; applied after 59 ms; tables() now ttlValue=1 ttlUnit=DAY
|
||
SET TTL 0 DAYS : seqTxn 9->10; applied after 25 ms; tables() now ttlValue=0 ttlUnit=HOUR
|
||
SET TTL 3 DAYS : seqTxn 10->11; applied after 60 ms; tables() now ttlValue=3 ttlUnit=DAY
|
||
after inserting now-20d and now into a TTL 7 DAYS table:
|
||
new | 2026-10-11T05:03:03
|
||
Q5c TTL reference clock (future-dated row):
|
||
after inserting a now+30d row, then one more ordinary commit:
|
||
d-3 | 2026-10-08 now | 2026-10-11 now+1h | 2026-10-11 d+30 | 2026-11-10
|
||
```
|
||
|
||
Image `conf/server.conf`: `#cairo.ttl.use.wall.clock=true` — "when true (default), TTL enforcement uses wall clock
|
||
time to prevent accidental data loss when future timestamps are inserted".
|
||
|
||
**Decision:**
|
||
- Create syntax: **`… TIMESTAMP(ts) PARTITION BY DAY TTL n DAYS WAL DEDUP UPSERT KEYS(…)`** — TTL goes **between
|
||
`PARTITION BY` and `WAL`**. Alter: **`ALTER TABLE t SET TTL n DAYS`**; remove: **`ALTER TABLE t SET TTL 0 DAYS`**.
|
||
`n` cannot be a parameter; it is the validated integer `RetentionDays`, formatted with `InvariantCulture`.
|
||
- Metadata: `SELECT ttlValue, ttlUnit FROM tables() WHERE table_name = $1`. **QuestDB normalises the unit**: 7 days
|
||
shows `1 WEEK`, 14 days `2 WEEK`, 30 days `30 DAY`, none `0 HOUR`. The bootstrapper compares in hours
|
||
(`HOUR`=1, `DAY`=24, `WEEK`=168; any other unit counts as different) and issues `ALTER` only when they differ.
|
||
- `ALTER … SET TTL` is a WAL transaction: `tables()` shows the old value until it is applied (25–100 ms). Do not
|
||
re-read to verify without the Q4 wait.
|
||
- Partitions older than TTL are dropped on commit. With the default wall-clock mode a future-dated sample does **not**
|
||
expire current data.
|
||
- TTL cannot go on the `PARTITION BY YEAR` `hist_tag` table; it is never needed there.
|
||
|
||
## Q6 — timestamp precision
|
||
|
||
```
|
||
raw sent 2026-03-01T12:00:00.1234567(Utc) -> read 2026-03-01T12:00:00.1234560 micros=1772366400123456 exact=False
|
||
raw9 sent 2026-03-01T12:00:01.1234569(Utc) -> read 2026-03-01T12:00:01.1234560 micros=1772366401123456 exact=False
|
||
pre sent 2026-03-01T12:01:00.1234560(Utc) -> read 2026-03-01T12:01:00.1234560 micros=1772366460123456 exact=True
|
||
```
|
||
|
||
**Decision:** Npgsql **truncates** sub-µs ticks (…1234567 → …123456; not rounded). That is truncation **toward zero
|
||
relative to 2000-01-01**, so before 2000 it rounds *up* (Q13). A pre-truncated (`ticks - ticks % 10`) value
|
||
round-trips exactly. This confirms R6: always `Round` first.
|
||
|
||
## Q7 — concurrent `CREATE TABLE IF NOT EXISTS`
|
||
|
||
```
|
||
Q7 concurrent CREATE TABLE IF NOT EXISTS (4 connections x 20 rounds)
|
||
errors: none
|
||
plain CREATE on existing: SqlState=00000 MessageText='table already exists'
|
||
```
|
||
|
||
(An earlier run with 2 connections × 20 rounds also had no errors.) **Decision:** `IF NOT EXISTS` was race-free here.
|
||
The S11 catch rule stays as defence: catch a `PostgresException` whose `MessageText` contains **`already exists`**,
|
||
then re-check with `SELECT count() FROM tables() WHERE table_name = $1`, and rethrow when the table is not there. Never
|
||
key on SQLSTATE (it is always `00000`).
|
||
|
||
## Q8 — bucket statistics
|
||
|
||
Seed (3 × 10 s buckets from `start = t0 + 0.5 s`): bucket 0 holds Good 5, Uncertain 2, Bad 9, a usable String, Good 2
|
||
(a min tie with the Uncertain 2), a tombstone of 100, and Good 7 at the last µs of the bucket. Bucket 1 holds Good 1 at
|
||
the edge, an Empty `BadNoCommunication` and an Empty with Good status. Bucket 2 holds Bad 3 only. Another tag sits in
|
||
range.
|
||
|
||
```
|
||
Simple family (GROUP BY alias):
|
||
cols: b, total_count, usable_count, good_count, bad_count, non_numeric_count, avg_value, min_value, max_value, sum_value
|
||
0 | 6 | 5 | 4 | 1 | 1 | 4 | 2 | 7 | 16
|
||
1 | 3 | 1 | 1 | 2 | 0 | 1 | 1 | 1 | 1
|
||
2 | 1 | 0 | 0 | 1 | 0 | NULL | NULL | NULL | NULL
|
||
[OK] simple GROUP BY <expression> (no subquery) [OK] simple implicit group-by
|
||
StartEnd form A (first()/last()):
|
||
0 | 6 | 5 | 4 | 1 | 1 | 1 | 5 | NULL | NULL | 00:00:00.5000000 | 1 | 7 | NULL | NULL | 00:00:10.4999990
|
||
1 | 3 | 1 | 1 | 2 | 0 | 1 | 1 | NULL | NULL | 00:00:10.5000000 | 1 | 1 | NULL | NULL | 00:00:10.5000000
|
||
2 | 1 | 0 | 0 | 1 | 0 | 0 | NULL | NULL | NULL | NULL | 0 | NULL | NULL | NULL | NULL
|
||
StartEnd form B (row_number window): identical rows [OK] filter-in-subquery [OK] extra predicate in ON
|
||
ActualTime (min): 0 ... | 2 | 00:00:01.5000000 1 ... | 1 | 00:00:10.5000000 2 ... | NULL | NULL
|
||
ActualTime (max): 0 ... | 7 | 00:00:10.4999990 1 ... | 1 | 00:00:10.5000000 2 ... | NULL | NULL
|
||
[OK] arg_min(ts, num_value) exists? 0 | 00:00:01.5 | 00:00:10.499999 1 | 00:00:10.5 | 00:00:10.5
|
||
randomized cross-check (60k rows over 3 days (4 partitions), 2 tags, 7 min buckets, random statuses/kinds/tombstones):
|
||
buckets=617 rowsInWindow=28423 mismatches=0; first()/last() 23 ms, window 21 ms, actual-time min+max 26 ms;
|
||
arg_min/arg_max (one stmt) 11 ms, tie-break mismatches vs earliest-ts rule = 0
|
||
```
|
||
|
||
Every hand-computed value matches: bucket 0 averages (5+2+2+7)/4 = 4, the tombstone is invisible, the sample one µs
|
||
before the edge stays in bucket 0, the edge sample is in bucket 1, and both Empty rows count as bad whatever their
|
||
status. The randomized cross-check compares counts, first/last usable times, sums and min/max actual times against a
|
||
client-side computation. It used a µs-aligned start that is not on a second boundary, and integer-rounded values to
|
||
force ties.
|
||
|
||
**Decision (the confirmed forms, with the SQL Server family split kept):**
|
||
- Classified subquery `s`. Status is stored as the **unsigned** uint in a LONG, so usable = `status < 2147483648` and
|
||
good = `status < 1073741824`, both also needing `kind <> 0`:
|
||
```sql
|
||
SELECT (cast(ts AS LONG) * 10 - $1) / $2 AS b, ts, kind, num_value, int_value, str_value,
|
||
CASE WHEN status < 2147483648 AND kind <> 0 THEN 1 ELSE 0 END AS u,
|
||
CASE WHEN status < 1073741824 AND kind <> 0 THEN 1 ELSE 0 END AS g
|
||
FROM {p}sample WHERE tag = $3 AND ts >= $4 AND ts < $5 AND NOT deleted
|
||
```
|
||
`$1` = window start in **ticks since the Unix epoch** (`start.Ticks - DateTime.UnixEpoch.Ticks`, `Bigint`); `$2` =
|
||
interval in ticks (`Bigint`); `$4`/`$5` = rounded window bounds. `cast(ts AS LONG)` is µs since 1970, so `*10` is
|
||
ticks. That is exact because both sides are already µs-aligned (R6). At year 9999 `µs*10` ≈ 2.5e18 < 9.2e18, so it
|
||
cannot overflow.
|
||
- Common counts: `count() AS total_count, sum(u) AS usable_count, sum(g) AS good_count, count() - sum(u) AS bad_count,
|
||
sum(CASE WHEN u = 1 AND kind IN (4, 5) THEN 1 ELSE 0 END) AS non_numeric_count`.
|
||
- **Simple:** `SELECT b, <counts>, avg(CASE WHEN u = 1 THEN num_value END) AS avg_value, min(…), max(…), sum(…) FROM (<s>)
|
||
GROUP BY b ORDER BY b`.
|
||
- **StartEnd:** the window form, which is the same shape as SQL Server:
|
||
`WITH s AS (<s>), st AS (SELECT b, <counts> FROM s GROUP BY b), f AS (SELECT b, kind, num_value, int_value,
|
||
str_value, ts, row_number() OVER (PARTITION BY b ORDER BY ts) AS rf, row_number() OVER (PARTITION BY b ORDER BY ts
|
||
DESC) AS rl FROM s WHERE u = 1) SELECT st.*, x.kind, x.num_value, x.int_value, x.str_value, x.ts, y.kind, …, y.ts
|
||
FROM st LEFT JOIN (SELECT * FROM f WHERE rf = 1) x ON x.b = st.b LEFT JOIN (SELECT * FROM f WHERE rl = 1) y ON
|
||
y.b = st.b ORDER BY st.b`. A bucket without a usable sample has NULL `ts` but **kind `0`** (BYTE has no NULL), so the
|
||
evaluator must key "no first/last" on the NULL time.
|
||
- **ActualTime:** `… r AS (SELECT b, num_value, ts, row_number() OVER (PARTITION BY b ORDER BY num_value ASC|DESC, ts
|
||
ASC) AS rn FROM s WHERE u = 1 AND num_value IS NOT NULL) SELECT st.*, r.num_value, r.ts FROM st LEFT JOIN (SELECT *
|
||
FROM r WHERE rn = 1) r ON r.b = st.b ORDER BY st.b`.
|
||
- **One query per node holds for every family.** The S11 two-statement fallback is not needed. `first()/last()` and
|
||
`arg_min/arg_max` also gave correct answers, but their tie order is not specified by an `ORDER BY`. The window form
|
||
states the earliest-timestamp tie-break explicitly, so it is the one to use.
|
||
- `SAMPLE BY` is not used.
|
||
|
||
## Q9 — AtTime neighbour lookup
|
||
|
||
200k rows (2 tags × 100k, ~1.5 s apart, 5 % tombstones). 102 request times, including one before the first sample,
|
||
one exact hit and one after the last sample.
|
||
|
||
```
|
||
form A SQL (one time): SELECT * FROM (SELECT 0 AS req, 0 AS dir, ts, kind, num_value, int_value, str_value, status FROM spike_q9
|
||
WHERE tag = $1 AND ts <= $2 AND NOT deleted ORDER BY ts DESC LIMIT 1) UNION ALL SELECT * FROM (SELECT 0 AS req, 1 AS dir, ts, ...
|
||
WHERE tag = $1 AND ts > $2 AND NOT deleted ORDER BY ts ASC LIMIT 1)
|
||
form A chunk= 1: 61 ms for 102 times, rows=202, mismatches=0
|
||
form A chunk= 10: 20 ms ... chunk= 25: 18 ms chunk= 50: 19 ms chunk= 60: 18 ms chunk= 63: 16 ms (all mismatches=0)
|
||
form A chunk= 64: FAIL 'SQL model is too complex to evaluate' (128 UNION ALL branches)
|
||
form A chunk=102: FAIL 'SQL model is too complex to evaluate' (204 UNION ALL branches)
|
||
form B: 1 ms for 102 times (prev only), rows=102, mismatches=0 [ASOF JOIN of a long_sequence(1) UNION ALL request set]
|
||
```
|
||
|
||
**Decision: form A, chunked `LIMIT 1` seeks, at most 50 request times (100 `UNION ALL` branches) per statement.**
|
||
The exact limit is **126 `UNION ALL` branches (63 request times) per statement**. A re-measure with single-seek
|
||
branches (review fix, below) showed 126 work and 127 fail. The form A runs above agree: 63 times (126 branches) work,
|
||
and 64 times (128 branches) fail. The literal `req` index and `dir` (0 = previous-or-equal, 1 = next) columns are
|
||
generated integers, not user input; `$1` is the tag and `$2..` the rounded times. It returns both neighbours in one
|
||
shape and costs ~0.2 ms per time. ASOF JOIN is faster, but it only gives the previous neighbour, so a second
|
||
seek form would still be needed for the next one. That does not justify two code paths at this scale.
|
||
|
||
## Q10 — long strings and SYMBOL capacity
|
||
|
||
```
|
||
[OK] 400-char tag (SYMBOL) + 400-char str_value (VARCHAR)
|
||
[OK] 5000-char str_value
|
||
30,000 distinct SYMBOL values inserted in 723 ms
|
||
count()=30002 | count_distinct(tag)=30002
|
||
400-char round trip: tag equal=True str equal=True lengths=400,400 (includes non-ASCII αβγ…)
|
||
tag | SYMBOL | symbolCached=True | symbolCapacity=16384
|
||
point lookup by tag among 30k symbols: count=1 0 ms
|
||
```
|
||
|
||
**Decision:** VARCHAR and SYMBOL take 400 characters (and VARCHAR takes 5,000). SYMBOL grows past its capacity hint
|
||
(16,384) without error, so 30k tags is fine. No `CAPACITY` clause is needed.
|
||
|
||
## Q11 — existing-timestamp lookup for a batch
|
||
|
||
```
|
||
[OK] range form: WHERE tag=$1 AND ts >= $2 AND ts <= $3 (tombstones included) -> 10 rows, incl. 00:00:08 deleted=True
|
||
[OK] IN-list form: ts IN ($2..$11) -> 4 rows (00, 06, 12, 18)
|
||
[OK] tombstone visible to range lookup -> 00:00:08 | True
|
||
```
|
||
|
||
**Decision:** both work with parameters. **Live writer (§2.2 step 2): the range form**
|
||
`SELECT ts FROM {p}sample WHERE tag = $1 AND ts >= $2 AND ts <= $3` (no `NOT deleted`, so tombstones count as present),
|
||
then a client-side set test. **Updater (sparse item times): the IN-list form** `SELECT ts, deleted FROM {p}sample WHERE
|
||
tag = $1 AND ts IN ($2, …)`, chunked at 1,000 times.
|
||
|
||
## Q12 — test cleanup and prefix listing
|
||
|
||
```
|
||
LIKE 't12_%' (note: _ is a wildcard) -> t12_sample, t12xsample <- WRONG: matches t12xsample
|
||
LIKE $1 with escaped _ ('t12\_%') -> t12_sample
|
||
starts_with(table_name, $1) -> t12_sample
|
||
left(table_name, 4) = $1 -> t12_sample
|
||
[OK] DROP TABLE IF EXISTS t12_sample (exists) [OK] DROP TABLE IF EXISTS t12_sample (already gone)
|
||
```
|
||
|
||
**Decision:** cleanup is `DROP TABLE IF EXISTS {name}`. List with **`SELECT table_name FROM tables() WHERE
|
||
starts_with(table_name, $1)`**. Never use `LIKE 'prefix%'`, because `_` in `hist_` is a wildcard.
|
||
|
||
## Q13 — binding extreme DateTimes
|
||
|
||
Seeds: 1970-01-01, 1990-05-05T00:00:00.123456, 2026-03-01, 2200-01-01. "Round" is R6's rule: `MinValue` → `+10 ticks`,
|
||
then `ticks - ticks % 10`.
|
||
|
||
```
|
||
DateTime.MinValue raw -> server sees µs -9222425352054775808 (Npgsql sends -infinity)
|
||
DateTime.MinValue Round -> 0001-01-01T00:00:00.0000010 (µs -62135596799999999)
|
||
DateTime.MaxValue raw -> server sees µs -9222425352054775809 (Npgsql sends +infinity)
|
||
DateTime.MaxValue Round -> 9999-12-31T23:59:59.9999990 (µs 253402300799999999)
|
||
0001-01-01 raw -> µs -9222425352054775808 (== MinValue -> -infinity)
|
||
1601-01-01 raw/Round -> 1601-01-01T00:00:00 (µs -11644473600000000)
|
||
9999-12-31T23:59:59.9999999 raw -> µs -9222425352054775809 (== MaxValue -> +infinity)
|
||
1969-12-31T23:59:59.9999995 raw -> 1970-01-01T00:00:00.0000000 (µs 0) Round -> 1969-12-31T23:59:59.9999990 (µs -1)
|
||
1990-05-05T00:00:00.1234567 raw -> 1990-05-05T00:00:00.1234570 Round -> ...1234560
|
||
1990-05-05T00:00:00.1234565 raw -> 1990-05-05T00:00:00.1234570 Round -> ...1234560
|
||
2026-03-01T00:00:00.0000005 raw -> 2026-03-01T00:00:00.0000000 Round -> same
|
||
range queries WHERE ts >= $1 AND ts < $2:
|
||
Min..Max raw count=0 <- silently empty
|
||
Min..Max Round count=4
|
||
0001..9999.9999999 Round count=4
|
||
1601..t0 Round count=2
|
||
1990.1234565..1990.1234575 raw count=0 <- seed .123456 missed (bound arrived as .123457)
|
||
1990.1234565..1990.1234575 Round count=1
|
||
inserting extreme designated timestamps:
|
||
[FAIL] insert ts=1969-12-31T23:59:59: PostgresException: 00000: designated timestamp before 1970-01-01 is not allowed
|
||
[FAIL] insert ts=1601-01-01: ... [FAIL] insert ts=Min+10 ticks: ...
|
||
[OK] insert ts=9999-12-31 Round -> 9999-12-31T23:59:59.9999990
|
||
```
|
||
|
||
**Decision:** R6's `BindTimestamp` is **confirmed necessary and sufficient**.
|
||
- A raw `DateTime.MinValue`/`MaxValue` (and anything equal to them) becomes ±infinity, and QuestDB stores that as a
|
||
near-`long.MinValue` value. A `[Min, Max)` range then matches **nothing**, with no error.
|
||
- A raw pre-2000 sub-µs value rounds **up**, which shifts a bound across a stored sample.
|
||
- With Round, every bound behaves.
|
||
- **New constraint:** a designated timestamp **before 1970-01-01** cannot be inserted. Reads with pre-1970 bounds work.
|
||
See the spec §2.2 ruling candidate.
|
||
|
||
## DDL and DeleteRaw paging (extra)
|
||
|
||
```
|
||
[OK] CREATE TABLE IF NOT EXISTS spike_hist_sample (...) TIMESTAMP(ts) PARTITION BY DAY TTL 30 DAYS WAL DEDUP UPSERT KEYS(ts, tag)
|
||
[OK] CREATE TABLE IF NOT EXISTS spike_hist_event (...) TIMESTAMP(ts) PARTITION BY DAY TTL 30 DAYS WAL DEDUP UPSERT KEYS(ts, event_id)
|
||
[OK] CREATE TABLE IF NOT EXISTS spike_hist_tag (...) TIMESTAMP(ts) PARTITION BY YEAR WAL DEDUP UPSERT KEYS(ts, name)
|
||
[OK] re-run (idempotent) x3
|
||
spike_hist_event | DAY | True | True | 30 | DAY
|
||
spike_hist_sample | DAY | True | True | 30 | DAY
|
||
spike_hist_tag | YEAR | True | True | 0 | HOUR
|
||
hist_tag upsert-by-name at ts=1970-01-01 (tagA twice, tagB once):
|
||
1970-01-01T00:00:00 | tagA | 4 | degF | second
|
||
1970-01-01T00:00:00 | tagB | 1 | NULL | NULL
|
||
DeleteRaw paging (page = 1000 rows): 2500 live rows + 1 outside range
|
||
pages=3 tombstones=2500 in 103 ms
|
||
deleted=False 1 deleted=True 2500
|
||
```
|
||
|
||
DeleteRaw loop, as confirmed: `SELECT ts FROM {p}sample WHERE tag = $1 AND ts >= $2 AND ts < $3 AND NOT deleted ORDER BY
|
||
ts LIMIT 1000` → one 1,000-row tombstone INSERT → the Q4 wait → continue from `last ts + 10 ticks`. The wait matters:
|
||
without it the next page can re-read rows it just tombstoned.
|
||
|
||
## Review fixes (2026-10-11)
|
||
|
||
### Raw `LIMIT $n` with a bound parameter
|
||
|
||
`dotnet run -- limit` against 10.0.1, with 10 rows seeded in `[t0, t0+10 s)`. The SQL was `SELECT ts, num_value FROM
|
||
spike_lim WHERE tag = $1 AND ts >= $2 AND ts < $3 AND NOT deleted ORDER BY ts [DESC] LIMIT $4`:
|
||
|
||
```
|
||
rows=3 first=0 last=2 [OK] ASC LIMIT $4 (Integer 3)
|
||
rows=3 first=9 last=7 [OK] DESC LIMIT $4 (Integer 3)
|
||
rows=3 first=0 last=2 [OK] ASC LIMIT $4 (Bigint 3)
|
||
rows=3 first=9 last=7 [OK] DESC LIMIT $4 (Bigint 3)
|
||
rows=10 first=0 last=9 [OK] ASC LIMIT $4 (Integer int.MaxValue)
|
||
rows=10 first=9 last=0 [OK] DESC LIMIT $4 (Integer int.MaxValue)
|
||
rows=10 first=0 last=9 [OK] ASC LIMIT $4 (Bigint long.MaxValue)
|
||
rows=10 first=9 last=0 [OK] DESC LIMIT $4 (Bigint long.MaxValue)
|
||
rows=0 first=NULL last=NULL [OK] ASC LIMIT $4 (Integer 0)
|
||
rows=0 first=NULL last=NULL [OK] DESC LIMIT $4 (Integer 0)
|
||
```
|
||
|
||
**Decision:** a bound `LIMIT $n` works, with `NpgsqlDbType.Integer` (Int32) or `Bigint` (Int64). Use `Integer`, since
|
||
`top` is an `int`. R8's unbounded value `int.MaxValue` returns every row. `LIMIT 0` returns nothing, but R8 never sends
|
||
it. The inlined-literal fallback is not needed.
|
||
|
||
### Exact `UNION ALL` branch limit
|
||
|
||
`dotnet run -- union`. Each branch is `SELECT * FROM (SELECT k AS req, ts FROM spike_un WHERE tag = $1 AND ts <= $2 AND
|
||
NOT deleted ORDER BY ts DESC LIMIT 1)`:
|
||
|
||
```
|
||
[OK] 100 branches [OK] 120 branches [OK] 125 branches [OK] 126 branches
|
||
[FAIL] 127 branches: PostgresException: 00000: SQL model is too complex to evaluate
|
||
[FAIL] 128 branches / 129 / 130: same
|
||
```
|
||
|
||
**Decision:** the limit is **126 branches, which is 63 AtTime request times** (two branches per time). The chosen chunk
|
||
of **50 times (100 branches)** is unchanged.
|
||
|
||
## Rulings candidates raised by the spike
|
||
|
||
1. **Pre-1970 timestamps cannot be stored** (Q13). QuestDB rejects a designated timestamp before 1970-01-01, and the
|
||
whole multi-row INSERT fails. Proposed:
|
||
- the live writer drops such samples and logs one Warning per batch. Retrying cannot succeed, and sending the
|
||
sample would fail the tag's whole group into #538 quarantine;
|
||
- HistoryUpdate Insert/Replace/Update items before 1970 return `BadOutOfRange`;
|
||
- DeleteAtTime/DeleteRaw find nothing there and return `BadNoEntryExists`;
|
||
- reads are unaffected.
|
||
Conformance tests that seed pre-1970 data are skipped for QuestDB.
|
||
2. **`long.MinValue` is QuestDB's LONG NULL** (sentinel check). An Int64 sample of exactly `long.MinValue` would read
|
||
back `int_value` NULL, and `FromColumns` would then serve **0**. Proposed: the QuestDB read path repairs it. When
|
||
`kind = Int64`, `int_value IS NULL` and `num_value = -9.223372036854775808E18` (exact in a double), serve
|
||
`long.MinValue`. That is a local fix-up before the shared `FromColumns`, with no write-side change.
|
||
3. **TTL metadata is unit-normalised and applied asynchronously** (Q5). The bootstrapper compares in hours and does not
|
||
re-read straight after an `ALTER`.
|