Files

T

Joseph Doherty 27991434a0 spike(historian): T0 review fixes

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe

2026-10-11 01:10:31 -04:00

29 KiB

Raw Blame History

QuestDB historian backend — T0 feasibility spike

Status: run 2026-10-11 against a live container. Gitea #666. Spec: docs/plans/2026-10-10-questdb-historian-design.md (§2–§4 amended in place from this doc; every confirmed form there is marked T0-confirmed).

Fixture

Item	Value
Image	`questdb/questdb:10.0.1` — Docker Hub `tag_last_pushed` 2026-08-24T14:00:08Z (48 days before 2026-10-10)
Why that tag	Newest exact `x.y.z` tag ≥ 14 days older than 2026-10-10. Newer tags are `nightly` / `patch-` only. Floor for TTL is 8.2.3; 10.0.1 is well above it.
Build	`SELECT build()` → `Build Information: QuestDB 10.0.1, JDK 25.0.2, Commit Hash 7a391566ac827e0d8c269b91f2415c16b0a32a84`
Overlay	`docker-dev/docker-compose.questdb.yml` — container `otopcua-dev-questdb-1`, ports `18812:8812` (PG wire) and `19000:9000` (console/REST), named volume `otopcua-questdb-data` (explicit `name:` so it is not project-prefixed)
Up	`docker compose -f docker-dev/docker-compose.yml -f docker-dev/docker-compose.questdb.yml up -d --no-deps questdb` (a `--dry-run` showed only the volume + `otopcua-dev-questdb-1` being created; nothing else in the `otopcua-dev` project is touched)
Healthcheck	the image ships `/usr/bin/curl` (`command -v curl` → `/usr/bin/curl`; no `wget`), so `curl -fs http://localhost:9003/status` is kept. Container reached `healthy`.
Harness	throwaway net10.0 console app, Npgsql 10.0.3 (the repo's central pin, `Directory.Packages.props` line 97), in the session scratchpad `questdb-spike/` with its own `Directory.Build.props`, `Directory.Packages.props` and `nuget.config`. Nothing was added to the repo.

Connection string (confirmed)

Host=localhost;Port=18812;Username=admin;Password=quest;Database=qdb;ServerCompatibilityMode=NoTypeLoading

Inside the docker network the same string uses Host=questdb;Port=8812.

Q1 — keywords, parameter types, CLR types

  [OK]   open+SELECT 1 with 'Host=localhost;Port=18812;Username=admin;Password=quest;Database=qdb'
  [OK]   open+SELECT 1 with '...;ServerCompatibilityMode=NoTypeLoading'
  [OK]   open->close->reopen x5 on one data source (no 'No Reset On Close')
  [OK]   reopen after a parameterised INSERT (prepared-ish state)
  [FAIL] ts Utc + Timestamp: ArgumentException: Cannot write DateTime with Kind=UTC to PostgreSQL type 'timestamp without time zone', consider using 'timestamp with time zone'. ...
  [OK]   ts Utc + TimestampTz
  [OK]   ts Utc + inferred
  [OK]   ts Unspecified + Timestamp
  [FAIL] ts Local + TimestampTz: ArgumentException: Cannot write DateTime with Kind=Local to PostgreSQL type 'timestamp with time zone', only UTC is supported. ...
  [OK]   kind bound as Integer (int4)
  [OK]   tag bound as Text
  [OK]   NaN double insert
  [OK]   +Inf double insert
    cols: ts:timestamp without time zone/DateTime, tag:character varying/String, kind:smallint/Int16, num_value:double precision/Double,
          int_value:bigint/Int64, str_value:character varying/String, status:bigint/Int64, deleted:boolean/Boolean, server_ts:timestamp without time zone/DateTime
    2026-01-02T03:04:05.1234560(Unspecified) | ts Utc + TimestampTz | 3<Int16> | 1.5 | 7<Int64> | s | 2147483648<Int64> | False | 2026-10-11T04:56:27.1103710(Unspecified)
    ...
    2026-01-02T03:04:11.1234560(Unspecified) | nan | 0<Int16> | NULL | NULL | NULL | NULL | False | NULL
    2026-01-02T03:04:12.1234560(Unspecified) | inf | 0<Int16> | NULL | NULL | NULL | NULL | False | NULL
  integer arithmetic with $n:
    cols: ts::LONG - $1:bigint/Int64                 [OK] cast(ts AS LONG) - $1 (Bigint)
    cols: (ts::LONG * 10 - $1) / $2:bigint/Int64     [OK] (Bigint)  [OK] (Integer)

Sentinel NULLs (extra check):

    cols: ts, l:bigint/Int64, i:integer/Int32, b:smallint/Int16, z:boolean/Boolean, l = NULL:boolean
    2026-03-01T00:00:00 | NULL | NULL | 0 | False | True          <- l = long.MinValue, i = int.MinValue written
    2026-03-01T00:00:01 | -9223372036854775807 | -2147483647 | 0 | False | False
    2026-03-01T00:00:02 | NULL | NULL | 0 | False | True          <- BYTE/BOOLEAN bound as DBNull

Decisions

Both connection strings work. Keep ServerCompatibilityMode=NoTypeLoading: it skips the pg_type load, and every form in this doc was run with it on. No Reset On Close is not needed: open → close → reopen on one pooled NpgsqlDataSource works.
Timestamps are bound as NpgsqlDbType.TimestampTz with DateTimeKind.Utc. Timestamp + Utc throws in Npgsql before reaching the server. Unspecified + Timestamp also works, but one rule is enough.
Timestamps read back as DateTime with Kind=Unspecified, wall clock = UTC. The reader must DateTime.SpecifyKind(r.GetDateTime(i), DateTimeKind.Utc).
CLR types returned: BYTE → smallint / Int16 (GetInt16); SYMBOL → character varying / String; VARCHAR → String; LONG → Int64; INT → Int32; BOOLEAN → Boolean; TIMESTAMP → DateTime; DOUBLE → Double. count()/sum(int) → Int64.
Parameter types: BYTE ← Smallint (Integer also accepted); SYMBOL/VARCHAR ← Varchar (Text also accepted); LONG ← Bigint; BOOLEAN ← Boolean; DOUBLE ← Double; a NULL is DBNull.Value with the column's NpgsqlDbType.
$n in integer arithmetic: bind as NpgsqlDbType.Bigint (Integer also worked for small values; Bigint is the rule because tick offsets exceed int32).
DOUBLE NaN and ±Inf read back as NULL. LONG long.MinValue and INT int.MinValue are QuestDB's NULL sentinels and read back NULL. BYTE and BOOLEAN have no NULL: unset or DBNull reads 0 / false. So deleted defaults to false and kind of a never-written field is 0 (Empty).
Every server error carries SqlState = 00000. Classification cannot use SQLSTATE: a PostgresException is a server-side rejection (decide by MessageText), any other NpgsqlException/IOException/SocketException/timeout is transport.

Q2 — multi-row parameterised INSERT and batch size

  [OK]   1000 rows x 9 params = 9000 params      88 ms
  [OK]   2000 rows x 9 params = 18000 params    167 ms
  [OK]   3640 rows x 9 params = 32760 params    342 ms
  [FAIL] 3641 rows x 9 params = 32769 params: PostgresException: 00000: invalid parameter count [parameterCount=-32767, offset=225670
  [FAIL] 5000 rows x 9 params = 45000 params: PostgresException: 00000: invalid parameter count [parameterCount=-20536, ...
  [FAIL] 7282 rows x 9 params = 65538 params: NpgsqlException: A statement cannot have more than 65535 parameters
  20 x 1000-row statements: fails=0, 122 ms total
    count()=27251 | count_distinct(ts)=27251        expected rows = 27251

Decision: QuestDB reads the Bind parameter count as a signed int16, so the real limit is 32,767 parameters per statement, not Npgsql's 65,535. Batch size = 1,000 rows per INSERT … VALUES (…),(…) (9,000 parameters for hist_sample), which is reliable and far under the limit. DeleteRaw page size = 1,000 rows (see the DDL section).

Q3 — `DEDUP UPSERT KEYS`

  before:   2026-03-01T00:00:00 | a | 1 | False      2026-03-01T00:00:00 | b | 2 | False
  after re-insert of (t0,'a') as tombstone:
            2026-03-01T00:00:00 | a | NULL | True     2026-03-01T00:00:00 | b | 2 | False
  NOT deleted:  2026-03-01T00:00:00 | b | 2
  same key twice in one statement (10 then 20):  2026-03-01T00:00:01 | c | 20
  hist_event with event_id VARCHAR as a dedup key:
  [OK]   CREATE hist_event-shape with DEDUP UPSERT KEYS(ts, event_id) on VARCHAR
  [OK]   insert e1 'first', e2, then e1 'second'
    2026-03-01T00:00:00 | e1 | src1 | 500 | second | NULL
    2026-03-01T00:00:00 | e2 | src1 | 500 | other  | NULL

Decision: re-inserting a key replaces the whole row; a tombstone written at (ts, tag) hides the row from NOT deleted. Within one statement the LAST duplicate wins, so the writer's "first value wins" rule must be applied client-side before the INSERT (spec §2.2 step 1 already does this). event_id VARCHAR works as a dedup key, so the S4 fallback (no dedup on hist_event) is not needed.

Q4 — `wal_tables()` and the visibility wait

    cols: name:varchar, suspended:boolean, writerTxn:bigint, bufferedTxnSize:bigint, sequencerTxn:bigint,
          errorTag:varchar, errorMessage:varchar, memoryPressure:integer
    run  0: seqTxn before=0 after=1 visibleImmediately=False waitMs=13 visibleAfterWait=True
    run  1: seqTxn before=1 after=2 visibleImmediately=False waitMs=0 visibleAfterWait=True
    ...
    run 16: seqTxn before=16 after=17 visibleImmediately=False waitMs=6 visibleAfterWait=True
  lag ms: min=0 median=0 p95=6 max=13; visible-without-wait 17/20

(An earlier run: visible without waiting 13/20, max 6 ms.) Every autocommitted INSERT is exactly one sequencer transaction.

Decision (the form T3's QuestDbWalVisibility implements):

After the INSERT(s) return, on the same or any connection: SELECT sequencerTxn FROM wal_tables() WHERE name = $1 ($1 = the table name, Varchar) → target.
Poll SELECT writerTxn, suspended FROM wal_tables() WHERE name = $1 every ~10 ms until writerTxn >= target, or suspended is true, or VisibilityTimeout elapses.

A concurrent writer can only make target larger, which lengthens the wait slightly and never shortens it. The wait is needed: up to 7/20 inserts were not yet visible on the next query. The S11 key-poll fallback is not needed.

Q5 — TTL

  [FAIL] CREATE ... PARTITION BY DAY WAL DEDUP UPSERT KEYS(ts, tag) TTL 7 DAYS (TTL last): unexpected token [TTL]
  [OK]   CREATE ... PARTITION BY DAY TTL 7 DAYS WAL DEDUP UPSERT KEYS(ts, tag) (TTL before WAL)
    spike_q5 | DAY | True | True | ttlValue=1 | ttlUnit=WEEK
  [FAIL] ALTER TABLE spike_q5 SET TTL $1 DAYS (parameter): invalid syntax, should be <number> <unit> but was $1
  [FAIL] TTL 7 DAYS on PARTITION BY YEAR table: TTL value must be an integer multiple of the partition size (its time interval)
Q5b ALTER ... SET TTL visibility in tables()
    SET TTL 30 DAYS : seqTxn 6->7; applied after 100 ms; tables() now ttlValue=30 ttlUnit=DAY
    SET TTL 14 DAYS : seqTxn 7->8; applied after 94 ms; tables() now ttlValue=2 ttlUnit=WEEK
    SET TTL 1 DAY   : seqTxn 8->9; applied after 59 ms; tables() now ttlValue=1 ttlUnit=DAY
    SET TTL 0 DAYS  : seqTxn 9->10; applied after 25 ms; tables() now ttlValue=0 ttlUnit=HOUR
    SET TTL 3 DAYS  : seqTxn 10->11; applied after 60 ms; tables() now ttlValue=3 ttlUnit=DAY
  after inserting now-20d and now into a TTL 7 DAYS table:
    new | 2026-10-11T05:03:03
Q5c TTL reference clock (future-dated row):
  after inserting a now+30d row, then one more ordinary commit:
    d-3 | 2026-10-08   now | 2026-10-11   now+1h | 2026-10-11   d+30 | 2026-11-10

Image conf/server.conf: #cairo.ttl.use.wall.clock=true — "when true (default), TTL enforcement uses wall clock time to prevent accidental data loss when future timestamps are inserted".

Decision:

Create syntax: … TIMESTAMP(ts) PARTITION BY DAY TTL n DAYS WAL DEDUP UPSERT KEYS(…) — TTL goes between PARTITION BY and WAL. Alter: ALTER TABLE t SET TTL n DAYS; remove: ALTER TABLE t SET TTL 0 DAYS. n cannot be a parameter; it is the validated integer RetentionDays, formatted with InvariantCulture.
Metadata: SELECT ttlValue, ttlUnit FROM tables() WHERE table_name = $1. QuestDB normalises the unit: 7 days shows 1 WEEK, 14 days 2 WEEK, 30 days 30 DAY, none 0 HOUR. The bootstrapper compares in hours (HOUR=1, DAY=24, WEEK=168; any other unit counts as different) and issues ALTER only when they differ.
ALTER … SET TTL is a WAL transaction: tables() shows the old value until it is applied (25–100 ms). Do not re-read to verify without the Q4 wait.
Partitions older than TTL are dropped on commit. With the default wall-clock mode a future-dated sample does not expire current data.
TTL cannot go on the PARTITION BY YEAR hist_tag table; it is never needed there.

Q6 — timestamp precision

    raw   sent 2026-03-01T12:00:00.1234567(Utc) -> read 2026-03-01T12:00:00.1234560 micros=1772366400123456 exact=False
    raw9  sent 2026-03-01T12:00:01.1234569(Utc) -> read 2026-03-01T12:00:01.1234560 micros=1772366401123456 exact=False
    pre   sent 2026-03-01T12:01:00.1234560(Utc) -> read 2026-03-01T12:01:00.1234560 micros=1772366460123456 exact=True

Decision: Npgsql truncates sub-µs ticks (…1234567 → …123456; not rounded). That is truncation toward zero relative to 2000-01-01, so before 2000 it rounds up (Q13). A pre-truncated (ticks - ticks % 10) value round-trips exactly. This confirms R6: always Round first.

Q7 — concurrent `CREATE TABLE IF NOT EXISTS`

Q7 concurrent CREATE TABLE IF NOT EXISTS (4 connections x 20 rounds)
  errors: none
  plain CREATE on existing: SqlState=00000 MessageText='table already exists'

(An earlier run with 2 connections × 20 rounds also had no errors.) Decision: IF NOT EXISTS was race-free here. The S11 catch rule stays as defence: catch a PostgresException whose MessageText contains already exists, then re-check with SELECT count() FROM tables() WHERE table_name = $1, and rethrow when the table is not there. Never key on SQLSTATE (it is always 00000).

Q8 — bucket statistics

Seed (3 × 10 s buckets from start = t0 + 0.5 s): bucket 0 holds Good 5, Uncertain 2, Bad 9, a usable String, Good 2 (a min tie with the Uncertain 2), a tombstone of 100, and Good 7 at the last µs of the bucket. Bucket 1 holds Good 1 at the edge, an Empty BadNoCommunication and an Empty with Good status. Bucket 2 holds Bad 3 only. Another tag sits in range.

  Simple family (GROUP BY alias):
    cols: b, total_count, usable_count, good_count, bad_count, non_numeric_count, avg_value, min_value, max_value, sum_value
    0 | 6 | 5 | 4 | 1 | 1 | 4 | 2 | 7 | 16
    1 | 3 | 1 | 1 | 2 | 0 | 1 | 1 | 1 | 1
    2 | 1 | 0 | 0 | 1 | 0 | NULL | NULL | NULL | NULL
  [OK] simple GROUP BY <expression> (no subquery)   [OK] simple implicit group-by
  StartEnd form A (first()/last()):
    0 | 6 | 5 | 4 | 1 | 1 | 1 | 5 | NULL | NULL | 00:00:00.5000000 | 1 | 7 | NULL | NULL | 00:00:10.4999990
    1 | 3 | 1 | 1 | 2 | 0 | 1 | 1 | NULL | NULL | 00:00:10.5000000 | 1 | 1 | NULL | NULL | 00:00:10.5000000
    2 | 1 | 0 | 0 | 1 | 0 | 0 | NULL | NULL | NULL | NULL | 0 | NULL | NULL | NULL | NULL
  StartEnd form B (row_number window): identical rows   [OK] filter-in-subquery   [OK] extra predicate in ON
  ActualTime (min): 0 ... | 2 | 00:00:01.5000000    1 ... | 1 | 00:00:10.5000000    2 ... | NULL | NULL
  ActualTime (max): 0 ... | 7 | 00:00:10.4999990    1 ... | 1 | 00:00:10.5000000    2 ... | NULL | NULL
  [OK] arg_min(ts, num_value) exists?  0 | 00:00:01.5 | 00:00:10.499999    1 | 00:00:10.5 | 00:00:10.5
  randomized cross-check (60k rows over 3 days (4 partitions), 2 tags, 7 min buckets, random statuses/kinds/tombstones):
    buckets=617 rowsInWindow=28423 mismatches=0; first()/last() 23 ms, window 21 ms, actual-time min+max 26 ms;
    arg_min/arg_max (one stmt) 11 ms, tie-break mismatches vs earliest-ts rule = 0

Every hand-computed value matches: bucket 0 averages (5+2+2+7)/4 = 4, the tombstone is invisible, the sample one µs before the edge stays in bucket 0, the edge sample is in bucket 1, and both Empty rows count as bad whatever their status. The randomized cross-check compares counts, first/last usable times, sums and min/max actual times against a client-side computation. It used a µs-aligned start that is not on a second boundary, and integer-rounded values to force ties.

Decision (the confirmed forms, with the SQL Server family split kept):

Classified subquery s. Status is stored as the unsigned uint in a LONG, so usable = status < 2147483648 and good = status < 1073741824, both also needing kind <> 0:
```
SELECT (cast(ts AS LONG) * 10 - $1) / $2 AS b, ts, kind, num_value, int_value, str_value,
       CASE WHEN status < 2147483648 AND kind <> 0 THEN 1 ELSE 0 END AS u,
       CASE WHEN status < 1073741824 AND kind <> 0 THEN 1 ELSE 0 END AS g
FROM {p}sample WHERE tag = $3 AND ts >= $4 AND ts < $5 AND NOT deleted
```
$1 = window start in ticks since the Unix epoch (start.Ticks - DateTime.UnixEpoch.Ticks, Bigint); $2 = interval in ticks (Bigint); $4/$5 = rounded window bounds. cast(ts AS LONG) is µs since 1970, so *10 is ticks. That is exact because both sides are already µs-aligned (R6). At year 9999 µs*10 ≈ 2.5e18 < 9.2e18, so it cannot overflow.
Common counts: count() AS total_count, sum(u) AS usable_count, sum(g) AS good_count, count() - sum(u) AS bad_count, sum(CASE WHEN u = 1 AND kind IN (4, 5) THEN 1 ELSE 0 END) AS non_numeric_count.
Simple: SELECT b, <counts>, avg(CASE WHEN u = 1 THEN num_value END) AS avg_value, min(…), max(…), sum(…) FROM (<s>) GROUP BY b ORDER BY b.
StartEnd: the window form, which is the same shape as SQL Server: WITH s AS (<s>), st AS (SELECT b, <counts> FROM s GROUP BY b), f AS (SELECT b, kind, num_value, int_value, str_value, ts, row_number() OVER (PARTITION BY b ORDER BY ts) AS rf, row_number() OVER (PARTITION BY b ORDER BY ts DESC) AS rl FROM s WHERE u = 1) SELECT st.*, x.kind, x.num_value, x.int_value, x.str_value, x.ts, y.kind, …, y.ts FROM st LEFT JOIN (SELECT * FROM f WHERE rf = 1) x ON x.b = st.b LEFT JOIN (SELECT * FROM f WHERE rl = 1) y ON y.b = st.b ORDER BY st.b. A bucket without a usable sample has NULL ts but kind 0 (BYTE has no NULL), so the evaluator must key "no first/last" on the NULL time.
ActualTime: … r AS (SELECT b, num_value, ts, row_number() OVER (PARTITION BY b ORDER BY num_value ASC|DESC, ts ASC) AS rn FROM s WHERE u = 1 AND num_value IS NOT NULL) SELECT st.*, r.num_value, r.ts FROM st LEFT JOIN (SELECT * FROM r WHERE rn = 1) r ON r.b = st.b ORDER BY st.b.
One query per node holds for every family. The S11 two-statement fallback is not needed. first()/last() and arg_min/arg_max also gave correct answers, but their tie order is not specified by an ORDER BY. The window form states the earliest-timestamp tie-break explicitly, so it is the one to use.
SAMPLE BY is not used.

Q9 — AtTime neighbour lookup

200k rows (2 tags × 100k, ~1.5 s apart, 5 % tombstones). 102 request times, including one before the first sample, one exact hit and one after the last sample.

    form A SQL (one time): SELECT * FROM (SELECT 0 AS req, 0 AS dir, ts, kind, num_value, int_value, str_value, status FROM spike_q9
      WHERE tag = $1 AND ts <= $2 AND NOT deleted ORDER BY ts DESC LIMIT 1) UNION ALL SELECT * FROM (SELECT 0 AS req, 1 AS dir, ts, ...
      WHERE tag = $1 AND ts > $2 AND NOT deleted ORDER BY ts ASC LIMIT 1)
    form A chunk=  1:   61 ms for 102 times, rows=202, mismatches=0
    form A chunk= 10:   20 ms ...   chunk= 25: 18 ms   chunk= 50: 19 ms   chunk= 60: 18 ms   chunk= 63: 16 ms   (all mismatches=0)
    form A chunk= 64: FAIL 'SQL model is too complex to evaluate' (128 UNION ALL branches)
    form A chunk=102: FAIL 'SQL model is too complex to evaluate' (204 UNION ALL branches)
    form B: 1 ms for 102 times (prev only), rows=102, mismatches=0     [ASOF JOIN of a long_sequence(1) UNION ALL request set]

Decision: form A, chunked LIMIT 1 seeks, at most 50 request times (100 UNION ALL branches) per statement. The exact limit is 126 UNION ALL branches (63 request times) per statement. A re-measure with single-seek branches (review fix, below) showed 126 work and 127 fail. The form A runs above agree: 63 times (126 branches) work, and 64 times (128 branches) fail. The literal req index and dir (0 = previous-or-equal, 1 = next) columns are generated integers, not user input; $1 is the tag and $2.. the rounded times. It returns both neighbours in one shape and costs ~0.2 ms per time. ASOF JOIN is faster, but it only gives the previous neighbour, so a second seek form would still be needed for the next one. That does not justify two code paths at this scale.

Q10 — long strings and SYMBOL capacity

  [OK]   400-char tag (SYMBOL) + 400-char str_value (VARCHAR)
  [OK]   5000-char str_value
    30,000 distinct SYMBOL values inserted in 723 ms
    count()=30002 | count_distinct(tag)=30002
    400-char round trip: tag equal=True str equal=True lengths=400,400      (includes non-ASCII αβγ…)
    tag | SYMBOL | symbolCached=True | symbolCapacity=16384
    point lookup by tag among 30k symbols: count=1 0 ms

Decision: VARCHAR and SYMBOL take 400 characters (and VARCHAR takes 5,000). SYMBOL grows past its capacity hint (16,384) without error, so 30k tags is fine. No CAPACITY clause is needed.

Q11 — existing-timestamp lookup for a batch

  [OK]   range form: WHERE tag=$1 AND ts >= $2 AND ts <= $3 (tombstones included)   -> 10 rows, incl. 00:00:08 deleted=True
  [OK]   IN-list form: ts IN ($2..$11)                                                -> 4 rows (00, 06, 12, 18)
  [OK]   tombstone visible to range lookup                                            -> 00:00:08 | True

Decision: both work with parameters. Live writer (§2.2 step 2): the range form SELECT ts FROM {p}sample WHERE tag = $1 AND ts >= $2 AND ts <= $3 (no NOT deleted, so tombstones count as present), then a client-side set test. Updater (sparse item times): the IN-list form SELECT ts, deleted FROM {p}sample WHERE tag = $1 AND ts IN ($2, …), chunked at 1,000 times.

Q12 — test cleanup and prefix listing

  LIKE 't12_%' (note: _ is a wildcard)  -> t12_sample, t12xsample      <- WRONG: matches t12xsample
  LIKE $1 with escaped _ ('t12\_%')     -> t12_sample
  starts_with(table_name, $1)           -> t12_sample
  left(table_name, 4) = $1              -> t12_sample
  [OK] DROP TABLE IF EXISTS t12_sample (exists)   [OK] DROP TABLE IF EXISTS t12_sample (already gone)

Decision: cleanup is DROP TABLE IF EXISTS {name}. List with SELECT table_name FROM tables() WHERE starts_with(table_name, $1). Never use LIKE 'prefix%', because _ in hist_ is a wildcard.

Q13 — binding extreme DateTimes

Seeds: 1970-01-01, 1990-05-05T00:00:00.123456, 2026-03-01, 2200-01-01. "Round" is R6's rule: MinValue → +10 ticks, then ticks - ticks % 10.

    DateTime.MinValue            raw   -> server sees µs -9222425352054775808   (Npgsql sends -infinity)
    DateTime.MinValue            Round -> 0001-01-01T00:00:00.0000010           (µs -62135596799999999)
    DateTime.MaxValue            raw   -> server sees µs -9222425352054775809   (Npgsql sends +infinity)
    DateTime.MaxValue            Round -> 9999-12-31T23:59:59.9999990           (µs 253402300799999999)
    0001-01-01                   raw   -> µs -9222425352054775808 (== MinValue -> -infinity)
    1601-01-01                   raw/Round -> 1601-01-01T00:00:00              (µs -11644473600000000)
    9999-12-31T23:59:59.9999999  raw   -> µs -9222425352054775809 (== MaxValue -> +infinity)
    1969-12-31T23:59:59.9999995  raw   -> 1970-01-01T00:00:00.0000000 (µs 0)   Round -> 1969-12-31T23:59:59.9999990 (µs -1)
    1990-05-05T00:00:00.1234567  raw   -> 1990-05-05T00:00:00.1234570          Round -> ...1234560
    1990-05-05T00:00:00.1234565  raw   -> 1990-05-05T00:00:00.1234570          Round -> ...1234560
    2026-03-01T00:00:00.0000005  raw   -> 2026-03-01T00:00:00.0000000          Round -> same
  range queries WHERE ts >= $1 AND ts < $2:
    Min..Max raw                                     count=0               <- silently empty
    Min..Max Round                                   count=4
    0001..9999.9999999 Round                         count=4
    1601..t0 Round                                   count=2
    1990.1234565..1990.1234575 raw                   count=0               <- seed .123456 missed (bound arrived as .123457)
    1990.1234565..1990.1234575 Round                 count=1
  inserting extreme designated timestamps:
  [FAIL] insert ts=1969-12-31T23:59:59: PostgresException: 00000: designated timestamp before 1970-01-01 is not allowed
  [FAIL] insert ts=1601-01-01: ...  [FAIL] insert ts=Min+10 ticks: ...
  [OK]   insert ts=9999-12-31 Round  -> 9999-12-31T23:59:59.9999990

Decision: R6's BindTimestamp is confirmed necessary and sufficient.

A raw DateTime.MinValue/MaxValue (and anything equal to them) becomes ±infinity, and QuestDB stores that as a near-long.MinValue value. A [Min, Max) range then matches nothing, with no error.
A raw pre-2000 sub-µs value rounds up, which shifts a bound across a stored sample.
With Round, every bound behaves.
New constraint: a designated timestamp before 1970-01-01 cannot be inserted. Reads with pre-1970 bounds work. See the spec §2.2 ruling candidate.

DDL and DeleteRaw paging (extra)

  [OK] CREATE TABLE IF NOT EXISTS spike_hist_sample (...) TIMESTAMP(ts) PARTITION BY DAY TTL 30 DAYS WAL DEDUP UPSERT KEYS(ts, tag)
  [OK] CREATE TABLE IF NOT EXISTS spike_hist_event (...)  TIMESTAMP(ts) PARTITION BY DAY TTL 30 DAYS WAL DEDUP UPSERT KEYS(ts, event_id)
  [OK] CREATE TABLE IF NOT EXISTS spike_hist_tag (...)    TIMESTAMP(ts) PARTITION BY YEAR WAL DEDUP UPSERT KEYS(ts, name)
  [OK] re-run (idempotent) x3
    spike_hist_event  | DAY  | True | True | 30 | DAY
    spike_hist_sample | DAY  | True | True | 30 | DAY
    spike_hist_tag    | YEAR | True | True | 0  | HOUR
  hist_tag upsert-by-name at ts=1970-01-01 (tagA twice, tagB once):
    1970-01-01T00:00:00 | tagA | 4 | degF | second
    1970-01-01T00:00:00 | tagB | 1 | NULL | NULL
  DeleteRaw paging (page = 1000 rows): 2500 live rows + 1 outside range
    pages=3 tombstones=2500 in 103 ms
    deleted=False 1    deleted=True 2500

DeleteRaw loop, as confirmed: SELECT ts FROM {p}sample WHERE tag = $1 AND ts >= $2 AND ts < $3 AND NOT deleted ORDER BY ts LIMIT 1000 → one 1,000-row tombstone INSERT → the Q4 wait → continue from last ts + 10 ticks. The wait matters: without it the next page can re-read rows it just tombstoned.

Review fixes (2026-10-11)

Raw `LIMIT $n` with a bound parameter

dotnet run -- limit against 10.0.1, with 10 rows seeded in [t0, t0+10 s). The SQL was SELECT ts, num_value FROM spike_lim WHERE tag = $1 AND ts >= $2 AND ts < $3 AND NOT deleted ORDER BY ts [DESC] LIMIT $4:

    rows=3 first=0 last=2     [OK]   ASC LIMIT $4 (Integer 3)
    rows=3 first=9 last=7     [OK]   DESC LIMIT $4 (Integer 3)
    rows=3 first=0 last=2     [OK]   ASC LIMIT $4 (Bigint 3)
    rows=3 first=9 last=7     [OK]   DESC LIMIT $4 (Bigint 3)
    rows=10 first=0 last=9    [OK]   ASC LIMIT $4 (Integer int.MaxValue)
    rows=10 first=9 last=0    [OK]   DESC LIMIT $4 (Integer int.MaxValue)
    rows=10 first=0 last=9    [OK]   ASC LIMIT $4 (Bigint long.MaxValue)
    rows=10 first=9 last=0    [OK]   DESC LIMIT $4 (Bigint long.MaxValue)
    rows=0 first=NULL last=NULL  [OK]   ASC LIMIT $4 (Integer 0)
    rows=0 first=NULL last=NULL  [OK]   DESC LIMIT $4 (Integer 0)

Decision: a bound LIMIT $n works, with NpgsqlDbType.Integer (Int32) or Bigint (Int64). Use Integer, since top is an int. R8's unbounded value int.MaxValue returns every row. LIMIT 0 returns nothing, but R8 never sends it. The inlined-literal fallback is not needed.

Exact `UNION ALL` branch limit

dotnet run -- union. Each branch is SELECT * FROM (SELECT k AS req, ts FROM spike_un WHERE tag = $1 AND ts <= $2 AND NOT deleted ORDER BY ts DESC LIMIT 1):

    [OK]   100 branches   [OK]   120 branches   [OK]   125 branches   [OK]   126 branches
    [FAIL] 127 branches: PostgresException: 00000: SQL model is too complex to evaluate
    [FAIL] 128 branches / 129 / 130: same

Decision: the limit is 126 branches, which is 63 AtTime request times (two branches per time). The chosen chunk of 50 times (100 branches) is unchanged.

Rulings candidates raised by the spike

Pre-1970 timestamps cannot be stored (Q13). QuestDB rejects a designated timestamp before 1970-01-01, and the whole multi-row INSERT fails. Proposed:
- the live writer drops such samples and logs one Warning per batch. Retrying cannot succeed, and sending the sample would fail the tag's whole group into #538 quarantine;
- HistoryUpdate Insert/Replace/Update items before 1970 return BadOutOfRange;
- DeleteAtTime/DeleteRaw find nothing there and return BadNoEntryExists;
- reads are unaffected. Conformance tests that seed pre-1970 data are skipped for QuestDB.
long.MinValue is QuestDB's LONG NULL (sentinel check). An Int64 sample of exactly long.MinValue would read back int_value NULL, and FromColumns would then serve 0. Proposed: the QuestDB read path repairs it. When kind = Int64, int_value IS NULL and num_value = -9.223372036854775808E18 (exact in a double), serve long.MinValue. That is a local fix-up before the shared FromColumns, with no write-side change.
TTL metadata is unit-normalised and applied asynchronously (Q5). The bootstrapper compares in hours and does not re-read straight after an ALTER.

29 KiB Raw Blame History Unescape Escape