lmxopcua/docs/plans/2026-10-10-questdb-historian-design.md

QuestDB historian backend — design

Status: approved section by section on 2026-10-10. Gitea #666. As-built pass (T8, 2026-10-11): every section below was re-checked against the merged code; where the two disagreed the text was corrected inline and marked (as built). User documentation: docs/Historian.md §"QuestDB backend". Builds on: docs/plans/2026-10-09-pluggable-historian-design.md, which defines the provider seam, typed values, the HistoryUpdate service and the SQL Server backend, all merged to master as 8ec7a83c.

0. Decisions (user, 2026-10-10)

Topic	Decision
HistoryUpdate deletes	Soft delete. Rows carry a deleted flag. A delete writes a tombstone at the same (tag, ts). Every read and aggregate excludes tombstones. All five HistoryUpdate operations are supported.
Write path	Postgres wire protocol only, via Npgsql. It is already centrally pinned at 10.0.3. No ILP, no new NuGet package.
Retention	Included. RetentionDays is applied as table TTL. QuestDB drops expired day partitions itself.
Approach	B: shared dialect-free core plus per-backend SQL. The pure logic moves out of the SQL Server backend into Driver.Historian.Sql.Common. One conformance suite runs against both backends.
Timestamps	Microseconds. Postgres-wire timestamps are microsecond resolution, and real source timestamps are at best milliseconds. Two samples less than 1 µs apart collapse to one, and the first wins — within a batch, and also against an already-stored row (the existence check runs on the rounded time). This is documented.

Out of scope:

• event HistoryUpdate;
• Part 11 modified values (#668);
• time-weighted aggregates (#671);
• ILP ingest;
• QuestDB authentication beyond the connection string;
• any change to the Gateway backend.

1. Components

1.1 ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common (new, no I/O)

This is extracted from Driver.Historian.SqlServer with no behaviour change:

• StoredSample. It holds the time, the value, the stored status. It classifies a sample as usable, good, bad or uncertain. It applies the quality-contract clamp: a NULL value carrying a non-Bad status is served as BadNoData. It also defines the info-bit masks (CodeBitsMask, SeverityBadBit, SeverityMask).
• AtTimeEvaluator. This is the former SqlServerHistorianDataSource.AtTimeValue. It covers interpolation and stepping, the outage rule (previous neighbour Bad ⇒ null plus that code; previous usable and next Bad ⇒ hold the previous value as UncertainDataSubNormal), stepping forward after the last sample, T > now ⇒ Uncertain, and masking & ~0x3 before the info bits are set.
• BucketRow + AggregateEvaluator. This is the former Evaluate. Each backend's SQL produces the same per-bucket statistics: total count, usable/good/bad/uncertain counts, numeric sum/min/max, first and last value and time, the time of the min and of the max, and so on. The shared evaluator maps a BucketRow to the OPC UA value and status for each of the 13 supported aggregates. Unsupported aggregates throw HistoryAggregateNotSupportedException.
• BucketPlan. It handles window normalisation (a reversed window is swapped and returned descending, as built today), interval 0 ⇒ one bucket, the 100,000-bucket cap, and tick arithmetic. It exposes the bucket start and width in ticks and in microseconds.
• HistorianColumnMapping. It maps a HistorianValueKind to the stored data-type code (the former SqlHistorianDataTypes.FromKind). It also maps a HistorianValue to its value columns (num/int/str) and back.
• HistoryUpdateResults. These are the per-item result rules for Insert, Replace, Update, DeleteAtTime and DeleteRaw given "did a live row exist". They include the NaN/Inf rule (rejected BadInvalidArgument on update, stored as BadOutOfRange Empty on the live write) and the rule rejecting an Empty value with a non-Bad status.
• Shared constants. The named status consts that both backends emit (BadOutOfRange and others) move into one parity-visible *StatusCodes class here. Backend-specific codes stay with their backend.

Driver.Historian.SqlServer then keeps only its T-SQL, the connection, the bootstrapper, the tag catalog and the seam classes, all calling the shared core.

1.2 ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb (new)

Class	Responsibility
QuestDbHistorianOptions	Binds ServerHistorian:QuestDb (§4) and validates it.
QuestDbConnection	Npgsql NpgsqlDataSource (pooled), command timeouts, transient-error classification, UTC timestamp reads.
QuestDbSchemaBootstrapper	Idempotent CREATE TABLE IF NOT EXISTS for the three tables, then applies TTL. It is concurrency-safe across two nodes starting together, and caches a failure for 5 s.
QuestDbHistorianValueWriter	IHistorianValueWriter for live samples (§2.2).
QuestDbTagCatalog	(as built) The only "is this tag known" authority (R7): a hist_tag row, positive results cached. Reads never create a tag; the live writer and HistoryUpdate Insert/Update do (Replace and the deletes do not). Re-inserting a known tag is skipped, because the dedup upsert would erase the unit and description the provisioner wrote.
QuestDbWalVisibility	(as built) The read-your-writes wait (§2.3), used by every write path.
QuestDbTimeClusters	(as built) Groups request times into ranges (gap > 1 min starts a new one) so existence reads are one range query per cluster.
QuestDbTables, QuestDbAggregateQueries	(as built) DDL / TTL helpers, and the push-down aggregate SQL (§3).
QuestDbTagProvisioner	IHistorianProvisioning: upserts hist_tag rows in chunks; (as built) a request without a unit or description keeps the stored one.
QuestDbAlarmHistorianWriter	IAlarmHistorianWriter: (as built) one INSERT per event (not batched), with the deterministic SHA-256 event_id. Outcomes: Ack; RetryPlease for transient and unclassified errors; PermanentFail only for a pre-1970 time or a server message naming an inconvertible / out-of-range / too-long / null value.
QuestDbHistorianDataSource	IHistorianDataSource (§3), with QuestDB SQL for raw, at-time, events and bucket statistics.
QuestDbHistorianDataUpdater	IHistorianDataUpdater (§2.3), the soft-delete updater.
QuestDbHistorianProvider	IHistorianBackendProvider: Name = "QuestDb", SupportedKinds = all six (FrozenSet), one shared connection and bootstrapper.

Host changes:

• HistorianBackendRegistry gains "QuestDb".
• ServerHistorianOptionsValidator validates the QuestDb section, gated on consumers, the same way it gates SqlServer.
• Host.csproj gets a ProjectReference.
• (as built) IHistorianDataSource gains the default member TimestampResolutionTicks (1; QuestDB 10), and the node manager's Raw paging steps off a drained timestamp by it (§3).
• (as built) The provider builds its shared pieces (connection, bootstrapper, catalog, WAL wait) on the first Create* call, not in the constructor, so a provider nobody consumes never fails on its options.

2. Schema and writes

2.1 Tables

QuestDB has no schemas, so the tables carry a configurable TablePrefix (default hist_, [A-Za-z0-9_]+). All three are WAL tables with DEDUP UPSERT KEYS, so re-sending a row with the same key replaces it.

T0-confirmed against questdb/questdb:10.0.1 (docs/plans/2026-10-10-questdb-spike.md §DDL, Q3, Q5, Q10). The TTL n DAYS clause goes between PARTITION BY and WAL. Placed after the dedup clause it fails with unexpected token [TTL]. It is emitted only when RetentionDays is set, as an inlined integer, because it cannot be a parameter.

CREATE TABLE IF NOT EXISTS hist_sample (
  ts TIMESTAMP, tag SYMBOL, kind BYTE, num_value DOUBLE, int_value LONG, str_value VARCHAR,
  status LONG, deleted BOOLEAN, server_ts TIMESTAMP
) TIMESTAMP(ts) PARTITION BY DAY [TTL n DAYS] WAL DEDUP UPSERT KEYS(ts, tag);

CREATE TABLE IF NOT EXISTS hist_event (
  ts TIMESTAMP, event_id VARCHAR, source SYMBOL, alarm_id VARCHAR, alarm_name VARCHAR, alarm_type_name VARCHAR,
  severity INT, event_kind VARCHAR, message VARCHAR, user_name VARCHAR, comment VARCHAR, received_utc TIMESTAMP
) TIMESTAMP(ts) PARTITION BY DAY [TTL n DAYS] WAL DEDUP UPSERT KEYS(ts, event_id);

CREATE TABLE IF NOT EXISTS hist_tag (
  ts TIMESTAMP, name SYMBOL, data_type BYTE, eng_unit VARCHAR, description VARCHAR, updated_utc TIMESTAMP
) TIMESTAMP(ts) PARTITION BY YEAR WAL DEDUP UPSERT KEYS(ts, name);  -- ts is always 1970-01-01, which makes this an upsert by name

hist_event maps hist.Event one-for-one:

• EventTimeUtc → ts, EventId → event_id, SourceName → source;
• AlarmId/AlarmName/AlarmTypeName → alarm_id/alarm_name/alarm_type_name;
• Severity → severity, EventKind → event_kind, Message → message;
• UserName → user_name, Comment → comment, ReceivedTimeUtc → received_utc.

event_id VARCHAR works as a dedup key (T0 Q3), so re-sending an event replaces it. The S4 fallback (no dedup on hist_event) is not used.

Column details (T0-confirmed):

• status holds the uint status code unsigned in a LONG (0..4294967295, never negative). So "severity Bad" is status >= 2147483648, not < 0 as in SQL Server's signed int.

• int_value also holds Booleans as 0/1, matching SQL Server.

• kind is HistorianValueKind in a BYTE.

• Read-back CLR types:

  Column type	Npgsql type name	CLR type / reader call
  BYTE	smallint	Int16 (GetInt16)
  SYMBOL, VARCHAR	character varying	String
  LONG	bigint	Int64
  INT	integer	Int32
  BOOLEAN	boolean	Boolean
  DOUBLE	double precision	Double
  TIMESTAMP	timestamp without time zone	DateTime, Kind=Unspecified; the reader applies SpecifyKind(…, Utc)

• NULL semantics:

  • DOUBLE NaN and ±Inf read back as NULL.
  • long.MinValue (LONG) and int.MinValue (INT) are QuestDB's NULL sentinels.
  • BYTE and BOOLEAN have no NULL: unset or DBNull reads 0 / false. So deleted defaults to false, and a LEFT-JOINed "no row" yields kind = 0. Absence must be detected on a NULL ts, not on kind.

• Ruling RC2 (as built): an Int64 sample of exactly long.MinValue stores int_value as NULL. The QuestDB read path repairs it before the shared FromColumns: kind = Int64 ∧ int_value IS NULL ∧ num_value = -2^63 ⇒ long.MinValue. That double is exact. Without the repair the sample would be served as 0.

• A designated timestamp before 1970-01-01 cannot be inserted (designated timestamp before 1970-01-01 is not allowed). See §2.2 (RC1).

• Tag names: SYMBOL and VARCHAR take 400 characters (non-ASCII included). SYMBOL grows past its capacity hint (16,384) with no error, and was tested with 30,000 distinct tags. No CAPACITY clause is needed.

2.2 Live writes (recorder) — insert only if absent, the same as SQL Server

1. Group the batch by tag and remove duplicates within the batch. The first value wins. A timestamp is rounded to microseconds before this step, so microsecond collisions also collapse.
2. For each tag, fetch the timestamps that already exist. (as built) The batch's times are grouped into clusters (QuestDbTimeClusters: a gap over 1 minute starts a new range; the live writer also caps a range at 1,000 times), with one range query per cluster, so one far-away outlier does not widen a single query over every stored row in between. Tombstones are included, so a re-sent outbox batch can never resurrect a deleted row or overwrite a HistoryUpdate change. T0-confirmed form (Q11): SELECT ts FROM {p}sample WHERE tag = $1 AND ts >= $2 AND ts <= $3, with no NOT deleted, followed by a client-side set test. The tag is registered in QuestDbTagCatalog first (a write creates an unknown tag).
3. Insert only the new rows with parameterised multi-row INSERT … VALUES (…),(…), at most 1,000 rows per statement (T0-confirmed Q2). QuestDB reads the Bind parameter count as a signed int16, so the hard limit is 32,767 parameters. (as built) The live writer binds 8 parameters per row (deleted is the literal false; 4,095 rows would fit), the updater 9 (3,640 rows). Within one statement a duplicate key keeps the last row, which is why step 1's "first wins" is done client-side.
4. Stored-value rules are shared with SQL Server:
   • NaN/±Inf, or any Empty with a non-Bad status, is stored Empty with BadOutOfRange or BadNoData.
   • Comms-loss outage markers are stored as given.
   • A tag name over 400 characters is a per-tag failure.
   • Ruling RC1 (T0 Q13): a sample timestamped before 1970-01-01 cannot be stored, because QuestDB rejects the whole INSERT. The writer drops such samples before the INSERT, logs one Warning per batch and counts them (DroppedPre1970); the call still succeeds for the rest. A retry could never succeed, and sending the sample would push the tag's whole group into the #538 quarantine.
5. Any SQL or transport failure makes the write return false, so the recorder retries and the #538 per-tag quarantine still applies. It never throws.

The recorder is Primary-only (#537), so step 2 does not race another live writer. The recorder waits for WAL visibility, bounded by VisibilityTimeout (plan R5), using the §2.3 wait.

Binding (T0-confirmed Q1/Q6/Q13):

• Every timestamp parameter is NpgsqlDbType.TimestampTz with DateTimeKind.Utc, after Round (as built: Unspecified is taken as UTC, Local is converted, then ticks are truncated to a multiple of 10). BindTimestamp(…, clampToEpoch: true) is used only for write binds of a designated timestamp; read bounds are never clamped (a pre-1970 bound is valid, and clamping it would move the bound across the epoch row). Timestamp + Utc throws in Npgsql. Npgsql truncates sub-µs ticks toward 2000-01-01, which rounds up before 2000. A raw DateTime.MinValue/MaxValue is sent as ±infinity, and a [Min, Max) range then silently matches nothing. R6's BindTimestamp (round; MinValue → +10 ticks; MaxValue → −9 ticks) fixes all three.

• The other parameter types:

  Column	NpgsqlDbType
  tag/str_value/other strings	Varchar
  kind/data_type	Smallint
  num_value	Double
  int_value/status	Bigint
  deleted	Boolean
  severity	Integer

  NULL is DBNull.Value with the column's type.

• Every QuestDB server error carries SQLSTATE 00000. QuestDbConnection's classification therefore treats a PostgresException as a server-side rejection, judged by MessageText. Any other NpgsqlException, IOException, SocketException or timeout is transport (transient).

2.3 HistoryUpdate (soft delete)

Every operation is an INSERT, and the dedup upsert performs the replace. QuestDB UPDATE and DELETE are never issued. "Exists" means a row that is present and not deleted.

Operation	Effect	Result
Insert	Write the row when none exists; a tombstone counts as absent, so Insert over a tombstone returns GoodEntryInserted (an explicit client insert may re-populate a deleted time; only the live writer's redelivery is barred, §2.2)	GoodEntryInserted / BadEntryExists
Replace	Overwrite the row when one exists	GoodEntryReplaced / BadNoEntryExists
Update	Write the row either way	GoodEntryReplaced or GoodEntryInserted
DeleteAtTime	Write a tombstone (deleted = true, value NULL) where a row exists	Good / BadNoEntryExists
DeleteRaw	Write a tombstone for every live row in the range	the same codes SQL Server returns for the same cases

Further rules:

• The per-item rules come from HistoryUpdateResults, shared with SQL Server, so the codes are identical.
• Item order is preserved, with exactly one code per item.
• Read-your-writes. After its inserts, the updater polls wal_tables() until the table's applied transaction reaches the sequencer transaction it produced, up to VisibilityTimeout. On a timeout the write is kept, the codes are reported as normal, and a Warning is logged.
• The visibility wait (QuestDbWalVisibility, T0-confirmed Q4) is used by the writer and the updater (R5):
  1. After the INSERT(s) return: SELECT sequencerTxn FROM wal_tables() WHERE name = $1 ($1 = the full table name, Varchar) → target. Each autocommitted INSERT is exactly one sequencer transaction.
  2. Poll SELECT writerTxn, suspended FROM wal_tables() WHERE name = $1 about every 10 ms until writerTxn >= target. Stop early, with a Warning, if suspended is true. Stop at VisibilityTimeout.
  • Measured: median 0 ms, p95 6 ms, max 13 ms. Up to 7/20 inserts were not visible to an immediate re-read, so the wait is required. A concurrent writer can only raise target. The key-poll fallback (S11) is not used.
• (as built) Existence checks for item times (Insert/Replace/Update/DeleteAtTime) are not IN lists: they use the T0 Q11 range form, one query per QuestDbTimeClusters cluster, SELECT ts FROM {p}sample WHERE tag = $1 AND ts >= $2 AND ts <= $3 AND NOT deleted, and the set test is client-side. The IN ($2, …) form this section originally proposed is not used.
• Unknown tag (as built): Insert and Update register the tag in QuestDbTagCatalog; Replace and the deletes do not (Replace ⇒ BadNoEntryExists, DeleteAtTime ⇒ BadNoEntryExists, DeleteRaw ⇒ BadNoData).
• Tombstone shape (as built): kind = Empty (0), value columns NULL, status = BadNoData, deleted = true.
• DeleteRaw (T0-confirmed), page size 1,000:
  1. SELECT ts FROM {p}sample WHERE tag = $1 AND ts >= $2 AND ts < $3 AND NOT deleted ORDER BY ts LIMIT 1000.
  2. Write one 1,000-row tombstone INSERT.
  3. Run the visibility wait.
  4. Continue from last ts + 10 ticks, until a page comes back empty. The wait between pages is required, or the next page re-reads rows just tombstoned. Measured: 2,500 rows, 3 pages, 103 ms.
• Ruling RC1 (T0 Q13): an Insert, Replace or Update item timestamped before 1970-01-01 returns BadOutOfRange, because QuestDB cannot store it (nothing is sent; a null or MinValue timestamp is still BadInvalidTimestamp, and as built a pre-1970 item that is also NaN reports BadOutOfRange, not BadInvalidArgument, because the pre-1970 filter runs first). DeleteAtTime there returns BadNoEntryExists; DeleteRaw finds nothing. A DeleteRaw window may start before 1970. Pre-1970 conformance seeds are skipped for QuestDB.
• Concurrency (as built). The updater serializes calls per tag, in-process only (64 striped async locks, held from the existence read through the visibility wait), so two concurrent Insert/Replace/Delete batches on one tag cannot both act on a stale read. It does not lock against the live writer or a second node. The existence checks read committed data, so a live recorder sample at the same timestamp still in flight in the WAL can race the updater, and two nodes writing the same new key concurrently can produce last-write-wins (QuestDB has no UPDLOCK; the Primary gate on the recorder makes it rare). Updates normally target past timestamps, so this residual is accepted and documented.

2.4 Retention

When RetentionDays is set, the bootstrapper applies TTL to hist_sample and hist_event, and re-applies it when the configured value changes. QuestDB then drops expired day partitions, tombstones included. When unset, data is kept forever. hist_tag never expires.

T0-confirmed (Q5):

• Create: … PARTITION BY DAY TTL n DAYS WAL … (§2.1).
• Change: ALTER TABLE {p}sample SET TTL n DAYS. Remove (RetentionDays unset while the table has a TTL): ALTER TABLE … SET TTL 0 DAYS.
• n cannot be a $ parameter (invalid syntax, should be <number> <unit> but was $1). It is the validated integer, formatted with InvariantCulture.
• TTL is not allowed on the PARTITION BY YEAR hist_tag table (TTL value must be an integer multiple of the partition size).
• TTL is enforced only on write commits (as built, N3). QuestDB looks for expired partitions when a write commits to the table, so a quiet table never drops anything; old partitions linger until the next write.
• Reading the current TTL: SELECT ttlValue, ttlUnit FROM tables() WHERE table_name = $1. QuestDB normalises the unit: 7 days reads 1 WEEK, 14 days 2 WEEK, 30 days 30 DAY, none 0 HOUR. The bootstrapper compares in hours (HOUR=1, DAY=24, WEEK=168; any other unit counts as different, RC3) and issues an ALTER only on a difference. A large RetentionDays that QuestDB normalises to MONTH/YEAR therefore re-ALTERs on every boot (harmless). AutoCreateSchema=false touches no TTL.
• ALTER … SET TTL is applied through the WAL, so tables() shows the old value for 25–100 ms. The bootstrapper does not re-read to verify.
• Expiry is checked against the wall clock: the image default is cairo.ttl.use.wall.clock=true. A future-dated sample does not expire current data (T0 Q5c).
• Table listing: SELECT table_name FROM tables() WHERE starts_with(table_name, $1). Never LIKE 'hist_%', because _ is a wildcard.
• Concurrent bootstrap: CREATE TABLE IF NOT EXISTS from 4 connections × 20 rounds produced no errors. The S11 defence stays: catch a PostgresException whose MessageText contains already exists, re-check tables(), and rethrow if the table is absent. Never key on SQLSTATE (always 00000).

3. Reads

Every query filters NOT deleted. Every value goes through StoredSample, so the quality clamp is the same as SQL Server's.

• Raw. A range on ts, ORDER BY ts [DESC], LIMIT @top (maxValuesPerNode == 0 ⇒ unbounded), at most top samples, ContinuationPoint: null; the node manager pages by re-reading from the last timestamp (plan R8). The dedup key forbids same-instant ties. Bounds follow the existing data-source contract: forward [start, end), reverse (end, start] descending, and (as built) start == end reads the sample exactly at that (rounded) time. (as built) The data source reports TimestampResolutionTicks = 10, and the node manager steps off the last drained timestamp by that amount (forward T + 10, reverse T − 10): a 1-tick step is truncated back to T, so a one-value-per-page forward read would loop forever. The rule is not "trim everything ≤ the last emitted", because the same paging serves the Gateway backend, where same-timestamp ties are legal.
  • T0-confirmed form (review fix): SELECT ts, kind, num_value, int_value, str_value, status FROM {p}sample WHERE tag = $1 AND ts >= $2 AND ts < $3 AND NOT deleted ORDER BY ts [DESC] LIMIT $4.
  • $4 is a bound parameter, NpgsqlDbType.Integer (Int32) or Bigint (Int64); both work on 10.0.1, ASC and DESC. Use Integer, since top is an int.
  • int.MaxValue (the R8 "unbounded" value) and long.MaxValue both return every row. LIMIT 0 returns no rows, but R8 never sends 0.
  • The inlined-literal fallback is not needed.
• AtTime (T0-confirmed Q9). Find the previous-or-equal neighbour and the next neighbour of each requested time with LIMIT 1 seeks, UNION ALL-ed, at most 50 request times (100 branches) per statement. The exact QuestDB 10.0.1 limit is 126 UNION ALL branches per statement (63 request times): 126 work, and 127 fail with SQL model is too complex to evaluate. So 50 times leaves a 26-branch margin. Per time k, with $1 = tag and $j = the rounded time:

  SELECT * FROM (SELECT k AS req, 0 AS dir, ts, kind, num_value, int_value, str_value, status FROM {p}sample
    WHERE tag = $1 AND ts <= $j AND NOT deleted ORDER BY ts DESC LIMIT 1)
  UNION ALL
  SELECT * FROM (SELECT k AS req, 1 AS dir, ts, kind, num_value, int_value, str_value, status FROM {p}sample
    WHERE tag = $1 AND ts > $j AND NOT deleted ORDER BY ts ASC LIMIT 1)
  

  k and dir are generated integer literals. Measured: 102 times in about 16–20 ms over 100k rows per tag, with no mismatches. ASOF JOIN was faster (1 ms), but it yields only the previous neighbour, so it was not chosen. AtTimeEvaluator then decides the result. (as built, M3) Each time is evaluated at its rounded value but the result is stamped with the time the client requested (UTC-normalised, not truncated), as on SQL Server, so answers match requests. Stored data is µs-only, so a sub-µs request adjacent to a sample is an exact hit.
• Events. Filtered by source and ts range [start, end), ordered (ts, event_id) ascending (reversed windows are swapped, as built today). (as built) With a bound, maxEvents + 1 rows are fetched; the extra row is dropped and becomes the continuation point.
• Processed. One query per node (T0-confirmed for every family; the S11 two-statement fallback is not needed).
  • Bucket index (T0-confirmed): (cast(ts AS LONG) * 10 - $1) / $2.
    • cast(ts AS LONG) is µs since 1970.
    • $1 = the rounded window start in ticks since the Unix epoch (start.Ticks − DateTime.UnixEpoch.Ticks).
    • $2 = the interval in ticks. Both are NpgsqlDbType.Bigint.
    • The result is exact because both sides are µs-aligned (R6), and it cannot overflow before year 9999.
  • (as built) The bucket timestamp returned for a Processed bucket is the rounded window start plus i × interval, so with a sub-µs request start it differs from SQL Server's label by the sub-µs remainder; values and membership are identical.
  • The query returns the full BucketRow statistics. Usable, good, bad and uncertain counts come from sum(CASE …) over the status column.
  • It does not use SAMPLE BY, whose alignment rules differ from ours. Computing the index ourselves keeps the bucket edges identical to SQL Server: start inclusive, end exclusive, a partial final bucket, interval 0, and the 100k cap.
  • The time of the min or max value (ActualTime family) comes from a row_number() window function in the same single statement: ORDER BY num_value ASC|DESC, ts ASC, with ties going to the earliest sample (T0-confirmed; the form is below). No second statement is used. AggregateEvaluator then produces the values.
  • Confirmed forms (T0 Q8). They mirror SqlAggregateQueries's three families and column order. They were checked against hand-computed seeds and a 60k-row randomized cross-check across 4 partitions: 617 buckets, 0 mismatches.
    • Classified subquery s:

      SELECT (cast(ts AS LONG) * 10 - $1) / $2 AS b, ts, kind, num_value, int_value, str_value,
             CASE WHEN status < 2147483648 AND kind <> 0 THEN 1 ELSE 0 END AS u,
             CASE WHEN status < 1073741824 AND kind <> 0 THEN 1 ELSE 0 END AS g
      FROM {p}sample WHERE tag = $3 AND ts >= $4 AND ts < $5 AND NOT deleted
      

    • Counts (ordinals 1–5 after b): count() AS total_count, sum(u) AS usable_count, sum(g) AS good_count, count() - sum(u) AS bad_count, sum(CASE WHEN u = 1 AND kind IN (4, 5) THEN 1 ELSE 0 END) AS non_numeric_count.
    • Simple: SELECT b, <counts>, avg(CASE WHEN u = 1 THEN num_value END) AS avg_value, min(CASE WHEN u = 1 THEN num_value END) AS min_value, max(…) AS max_value, sum(…) AS sum_value FROM (<s>) GROUP BY b ORDER BY b.
    • StartEnd:

      WITH s AS (<s>), st AS (SELECT b, <counts> FROM s GROUP BY b),
           f AS (SELECT b, kind, num_value, int_value, str_value, ts,
                        row_number() OVER (PARTITION BY b ORDER BY ts) AS rf,
                        row_number() OVER (PARTITION BY b ORDER BY ts DESC) AS rl FROM s WHERE u = 1)
      SELECT st.*, x.kind, x.num_value, x.int_value, x.str_value, x.ts, y.kind, y.num_value, y.int_value, y.str_value, y.ts
      FROM st LEFT JOIN (SELECT * FROM f WHERE rf = 1) x ON x.b = st.b
              LEFT JOIN (SELECT * FROM f WHERE rl = 1) y ON y.b = st.b ORDER BY st.b
      

    • ActualTime: the same s/st, then r AS (SELECT b, num_value, ts, row_number() OVER (PARTITION BY b ORDER BY num_value ASC|DESC, ts ASC) AS rn FROM s WHERE u = 1 AND num_value IS NOT NULL) SELECT st.*, r.num_value, r.ts FROM st LEFT JOIN (SELECT * FROM r WHERE rn = 1) r ON r.b = st.b ORDER BY st.b.
    • A bucket with no usable sample returns NULL times but kind = 0 (BYTE has no NULL). The mapper keys absence on the NULL ts.
    • first()/last() and arg_min/arg_max also returned correct values, but no ORDER BY fixes their tie order. The window forms state the earliest-timestamp tie-break explicitly and are the ones to use.
• Errors. A failed read fails only that node, under the same data-source contract as SQL Server.

4. Configuration (ServerHistorian:QuestDb)

Key	Default	Validation
ConnectionString	—	Required when Backend=QuestDb and a consumer is enabled. An Npgsql string. T0-confirmed: Host=questdb;Port=8812;Username=admin;Password=quest;Database=qdb;ServerCompatibilityMode=NoTypeLoading (the docker-dev overlay from the host: Host=localhost;Port=18812;…). ServerCompatibilityMode=NoTypeLoading is kept: it skips the pg_type load, and every confirmed form was run with it. The string also works without it. No Reset On Close is not needed, because pooled open → close → reopen works.
TablePrefix	hist_	^[A-Za-z0-9_]+$. Only checked when a consumer is enabled.
RetentionDays	unset	≥ 1 when set
CommandTimeout	00:00:30	> 0
AutoCreateSchema	true	—
VisibilityTimeout	00:00:05	≥ 0. 0 disables the wait.

The Gateway keys (Endpoint, ApiKey, UseTls) are not required for QuestDb, the same as for SqlServer.

(as built) Pool size. QuestDB's default limit is 64 PG-wire connections; Npgsql's default Maximum Pool Size is 100, and the read, write, update and alarm paths share one pool per node. Operators should set Maximum Pool Size in the connection string below the server limit, per node (a redundant pair shares one QuestDB). The validator checks the prefix only when a consumer is enabled, but the connection helper also refuses a malformed prefix (ArgumentException) when the first seam is built.

5. Testing

• T1 refactor gate. The SQL Server integration suite (118) and unit suite pass unchanged before and after the extraction. Each moved piece of logic is falsified once against its existing test.
• Conformance suite (tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Conformance, a class library of abstract xUnit tests). Each backend's integration project subclasses it and supplies a fixture and an IConformanceSeeder, which writes rows directly in that backend's schema.
  • It covers raw forward, reverse and paging; AtTime, including the outage, after-last and future rules; all 13 aggregates with the existing hand-computed literals; the quality contract; insert-if-absent writes and redelivery; the HistoryUpdate result codes; and over-long tag names.
  • Each legitimate backend difference is a named, documented override, never a silent skip. Microsecond rounding is the only expected one.
• QuestDB-only tests:
  • a tombstone hides the row and a redelivery does not resurrect it;
  • the read-your-writes wait;
  • TTL is applied, verified through tables() metadata;
  • concurrent bootstrap from two instances;
  • intra-batch microsecond collisions;
  • the hist_tag upsert;
  • transient-failure behaviour, writing against a stopped port;
  • (as built) pre-1970 handling (writer drop, BadOutOfRange, read bounds), the long.MinValue read repair, Insert over a tombstone, per-tag updater serialization, and a forward cap-1 paging regression against a real QuestDB.
  • Conformance differences are named overrides: pre-1970 seeds are skipped on QuestDB, and AtTime timestamp assertions compare Round(x) to the requested time (identity on SQL Server). The QuestDB fixture prunes idle pool connections after 1 s because of the 64-connection limit.
• Fixture. HISTORIAN_QUESTDB_TEST_CONNECTION; when it is absent, the tests skip. Each test class uses a unique throwaway TablePrefix (t_<guid>_) and drops its tables on dispose.
• Host. Registry and validator tests for Backend=QuestDb, beside the SqlServer ones.

6. Environment

• QuestDB is an optional overlay file, docker-dev/docker-compose.questdb.yml, so the default rig is unchanged. It defines the questdb service:
  • image questdb/questdb:10.0.1, pinned by T0 and pushed 2026-08-24;
  • container otopcua-dev-questdb-1;
  • started with docker compose -f docker-dev/docker-compose.yml -f docker-dev/docker-compose.questdb.yml up -d --no-deps questdb.
• It exposes PG wire on host port 18812 (→ 8812) and the web console/REST on host port 19000 (→ 9000), so it does not collide with AdminUI :9000/:9200.
• It uses the named volume otopcua-questdb-data.
• The central nodes gain ServerHistorian__QuestDb__ConnectionString (OTOPCUA_HISTORIAN_QUESTDB), defaulting to the questdb service.
• No new NuGet packages.

7. Live gate (T9, run by the controller)

On docker-dev, with Backend=QuestDb:

1. Only the Primary records, with real kinds and status, one row per value.
2. Raw and Processed HistoryRead return identical results through the raw and UNS NodeIds, and match a direct QuestDB query.
3. HistoryUpdate covers insert, duplicate, replace, replace-missing and delete-at-time, each confirmed by a read and by a direct query. After a delete, an outbox redelivery does not bring the row back. A WriteOperate-only key and an anonymous session are both denied.
4. On failover, recording continues at 1×.
5. Modbus device loss stores a BadCommunicationError Empty sample, then the BadNoCommunication outage marker, then Good on recovery.
6. TTL is visible in table metadata when RetentionDays is set.

Evidence goes in docs/plans/2026-10-10-questdb-historian-live-gate.md, and the rig is restored afterwards.

8. Work order

Wave	Task	Model
W0	T0 spike. Pin the image, add the compose service, and prove every QuestDB assumption here: Npgsql parameterised multi-row inserts, DEDUP UPSERT, wal_tables() visibility, TTL syntax, microsecond timestamps over Npgsql, the min/max-time query form, AtTime neighbour lookup, concurrent CREATE TABLE IF NOT EXISTS. Amend §2–§4 with the findings before W1.	controller + sonnet
W1	T1: extract Sql.Common, leaving SQL Server's behaviour unchanged	opus
W2	T2 conformance suite over SQL Server ‖ T3 QuestDB skeleton (options, connection, bootstrapper, fixture)	sonnet ‖ sonnet
W3	T4 writes, provisioning and alarms ‖ T5 reads and aggregates ‖ T6 updater with soft delete	sonnet ‖ opus ‖ sonnet
W4	T7 provider, registry and validator, plus the conformance suite run against QuestDB	sonnet
W5	T8 docs (Historian.md, Configuration.md, CLAUDE.md, docker-dev README), then a Fable whole-branch review, one fix wave, and T9 live gate	sonnet / fable / opus / controller