dohertj2/lmxopcua
1
0
Fork 0
Code Issues 32 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
c4b4b2e2bca5545db8cbafab3e48bb2acf60deaa
lmxopcua/docs/plans/2026-10-10-questdb-historian-design.md
T
Joseph Doherty 27991434a0 spike(historian): T0 review fixes 
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-11 01:10:31 -04:00

28 KiB
Raw Blame History

QuestDB historian backend — design

Status: approved section by section on 2026-10-10. Gitea #666. Builds on: docs/plans/2026-10-09-pluggable-historian-design.md, which defines the provider seam, typed values, the HistoryUpdate service and the SQL Server backend, all merged to master as 8ec7a83c.

0. Decisions (user, 2026-10-10)

Topic Decision
HistoryUpdate deletes Soft delete. Rows carry a deleted flag. A delete writes a tombstone at the same (tag, ts). Every read and aggregate excludes tombstones. All five HistoryUpdate operations are supported.
Write path Postgres wire protocol only, via Npgsql. It is already centrally pinned at 10.0.3. No ILP, no new NuGet package.
Retention Included. RetentionDays is applied as table TTL. QuestDB drops expired day partitions itself.
Approach B: shared dialect-free core plus per-backend SQL. The pure logic moves out of the SQL Server backend into Driver.Historian.Sql.Common. One conformance suite runs against both backends.
Timestamps Microseconds. Postgres-wire timestamps are microsecond resolution, and real source timestamps are at best milliseconds. Two samples less than 1 µs apart within a batch collapse to one, and the first wins. This is documented.

Out of scope:

  • event HistoryUpdate;
  • Part 11 modified values (#668);
  • time-weighted aggregates (#671);
  • ILP ingest;
  • QuestDB authentication beyond the connection string;
  • any change to the Gateway backend.

1. Components

1.1 ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common (new, no I/O)

This is extracted from Driver.Historian.SqlServer with no behaviour change:

  • StoredSample. It holds the time, the value, the stored status. It classifies a sample as usable, good, bad or uncertain. It applies the quality-contract clamp: a NULL value carrying a non-Bad status is served as BadNoData. It also defines the info-bit masks (CodeBitsMask, SeverityBadBit, SeverityMask).
  • AtTimeEvaluator. This is the former SqlServerHistorianDataSource.AtTimeValue. It covers interpolation and stepping, the outage rule (previous neighbour Bad ⇒ null plus that code; previous usable and next Bad ⇒ hold the previous value as UncertainDataSubNormal), stepping forward after the last sample, T > now ⇒ Uncertain, and masking & ~0x3 before the info bits are set.
  • BucketRow + AggregateEvaluator. This is the former Evaluate. Each backend's SQL produces the same per-bucket statistics: total count, usable/good/bad/uncertain counts, numeric sum/min/max, first and last value and time, the time of the min and of the max, and so on. The shared evaluator maps a BucketRow to the OPC UA value and status for each of the 13 supported aggregates. Unsupported aggregates throw HistoryAggregateNotSupportedException.
  • BucketPlan. It handles window normalisation (a reversed window is swapped and returned descending, as built today), interval 0 ⇒ one bucket, the 100,000-bucket cap, and tick arithmetic. It exposes the bucket start and width in ticks and in microseconds.
  • HistorianColumnMapping. It maps a HistorianValueKind to the stored data-type code (the former SqlHistorianDataTypes.FromKind). It also maps a HistorianValue to its value columns (num/int/str) and back.
  • HistoryUpdateResults. These are the per-item result rules for Insert, Replace, Update, DeleteAtTime and DeleteRaw given "did a live row exist". They include the NaN/Inf rule (rejected BadInvalidArgument on update, stored as BadOutOfRange Empty on the live write) and the rule rejecting an Empty value with a non-Bad status.
  • Shared constants. The named status consts that both backends emit (BadOutOfRange and others) move into one parity-visible *StatusCodes class here. Backend-specific codes stay with their backend.

Driver.Historian.SqlServer then keeps only its T-SQL, the connection, the bootstrapper, the tag catalog and the seam classes, all calling the shared core.

1.2 ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb (new)

Class Responsibility
QuestDbHistorianOptions Binds ServerHistorian:QuestDb (§4) and validates it.
QuestDbConnection Npgsql NpgsqlDataSource (pooled), command timeouts, transient-error classification, UTC timestamp reads.
QuestDbSchemaBootstrapper Idempotent CREATE TABLE IF NOT EXISTS for the three tables, then applies TTL. It is concurrency-safe across two nodes starting together, and caches a failure for 5 s.
QuestDbHistorianValueWriter IHistorianValueWriter for live samples (§2.2).
QuestDbTagProvisioner IHistorianProvisioning: upserts hist_tag rows.
QuestDbAlarmHistorianWriter IAlarmHistorianWriter: writes events with the deterministic SHA-256 event_id and the same classify rules as SQL Server (unknown ⇒ RetryPlease).
QuestDbHistorianDataSource IHistorianDataSource (§3), with QuestDB SQL for raw, at-time, events and bucket statistics.
QuestDbHistorianDataUpdater IHistorianDataUpdater (§2.3), the soft-delete updater.
QuestDbHistorianProvider IHistorianBackendProvider: Name = "QuestDb", SupportedKinds = all six (FrozenSet), one shared connection and bootstrapper.

Host changes:

  • HistorianBackendRegistry gains "QuestDb".
  • ServerHistorianOptionsValidator validates the QuestDb section, gated on consumers, the same way it gates SqlServer.
  • Host.csproj gets a ProjectReference.

2. Schema and writes

2.1 Tables

QuestDB has no schemas, so the tables carry a configurable TablePrefix (default hist_, [A-Za-z0-9_]+). All three are WAL tables with DEDUP UPSERT KEYS, so re-sending a row with the same key replaces it.

T0-confirmed against questdb/questdb:10.0.1 (docs/plans/2026-10-10-questdb-spike.md §DDL, Q3, Q5, Q10). The TTL n DAYS clause goes between PARTITION BY and WAL. Placed after the dedup clause it fails with unexpected token [TTL]. It is emitted only when RetentionDays is set, as an inlined integer, because it cannot be a parameter.

CREATE TABLE IF NOT EXISTS hist_sample (
  ts TIMESTAMP, tag SYMBOL, kind BYTE, num_value DOUBLE, int_value LONG, str_value VARCHAR,
  status LONG, deleted BOOLEAN, server_ts TIMESTAMP
) TIMESTAMP(ts) PARTITION BY DAY [TTL n DAYS] WAL DEDUP UPSERT KEYS(ts, tag);

CREATE TABLE IF NOT EXISTS hist_event (
  ts TIMESTAMP, event_id VARCHAR, source SYMBOL, alarm_id VARCHAR, alarm_name VARCHAR, alarm_type_name VARCHAR,
  severity INT, event_kind VARCHAR, message VARCHAR, user_name VARCHAR, comment VARCHAR, received_utc TIMESTAMP
) TIMESTAMP(ts) PARTITION BY DAY [TTL n DAYS] WAL DEDUP UPSERT KEYS(ts, event_id);

CREATE TABLE IF NOT EXISTS hist_tag (
  ts TIMESTAMP, name SYMBOL, data_type BYTE, eng_unit VARCHAR, description VARCHAR, updated_utc TIMESTAMP
) TIMESTAMP(ts) PARTITION BY YEAR WAL DEDUP UPSERT KEYS(ts, name);  -- ts is always 1970-01-01, which makes this an upsert by name

hist_event maps hist.Event one-for-one:

  • EventTimeUtc → ts, EventId → event_id, SourceName → source;
  • AlarmId/AlarmName/AlarmTypeName → alarm_id/alarm_name/alarm_type_name;
  • Severity → severity, EventKind → event_kind, Message → message;
  • UserName → user_name, Comment → comment, ReceivedTimeUtc → received_utc.

event_id VARCHAR works as a dedup key (T0 Q3), so re-sending an event replaces it. The S4 fallback (no dedup on hist_event) is not used.

Column details (T0-confirmed):

  • status holds the uint status code unsigned in a LONG (0..4294967295, never negative). So "severity Bad" is status >= 2147483648, not < 0 as in SQL Server's signed int.

  • int_value also holds Booleans as 0/1, matching SQL Server.

  • kind is HistorianValueKind in a BYTE.

  • Read-back CLR types:

    Column type Npgsql type name CLR type / reader call
    BYTE smallint Int16 (GetInt16)
    SYMBOL, VARCHAR character varying String
    LONG bigint Int64
    INT integer Int32
    BOOLEAN boolean Boolean
    DOUBLE double precision Double
    TIMESTAMP timestamp without time zone DateTime, Kind=Unspecified; the reader applies SpecifyKind(…, Utc)

  • NULL semantics:

    • DOUBLE NaN and ±Inf read back as NULL.
    • long.MinValue (LONG) and int.MinValue (INT) are QuestDB's NULL sentinels.
    • BYTE and BOOLEAN have no NULL: unset or DBNull reads 0 / false. So deleted defaults to false, and a LEFT-JOINed "no row" yields kind = 0. Absence must be detected on a NULL ts, not on kind.

  • Ruling candidate (T0): an Int64 sample of exactly long.MinValue stores int_value as NULL. The QuestDB read path repairs it before the shared FromColumns: kind = Int64 ∧ int_value IS NULL ∧ num_value = -2^63 ⇒ long.MinValue. That double is exact. Without the repair the sample would be served as 0.

  • A designated timestamp before 1970-01-01 cannot be inserted (designated timestamp before 1970-01-01 is not allowed). See §2.2.

  • Tag names: SYMBOL and VARCHAR take 400 characters (non-ASCII included). SYMBOL grows past its capacity hint (16,384) with no error, and was tested with 30,000 distinct tags. No CAPACITY clause is needed.

2.2 Live writes (recorder) — insert only if absent, the same as SQL Server

  1. Group the batch by tag and remove duplicates within the batch. The first value wins. A timestamp is rounded to microseconds before this step, so microsecond collisions also collapse.
  2. For each tag, fetch the timestamps that already exist in the batch's [min ts, max ts] range. Tombstones are included, so a re-sent outbox batch can never resurrect a deleted row or overwrite a HistoryUpdate change. T0-confirmed form (Q11): SELECT ts FROM {p}sample WHERE tag = $1 AND ts >= $2 AND ts <= $3, with no NOT deleted, followed by a client-side set test.
  3. Insert only the new rows with parameterised multi-row INSERT … VALUES (…),(…), at most 1,000 rows per statement (T0-confirmed Q2). QuestDB reads the Bind parameter count as a signed int16, so the hard limit is 32,767 parameters (3,640 nine-column rows). Within one statement a duplicate key keeps the last row, which is why step 1's "first wins" is done client-side.
  4. Stored-value rules are shared with SQL Server:
    • NaN/±Inf, or any Empty with a non-Bad status, is stored Empty with BadOutOfRange or BadNoData.
    • Comms-loss outage markers are stored as given.
    • A tag name over 400 characters is a per-tag failure.
    • Ruling candidate (T0 Q13): a sample timestamped before 1970-01-01 cannot be stored, because QuestDB rejects the whole INSERT. The writer drops such samples before the INSERT and logs one Warning per batch. A retry could never succeed, and sending the sample would push the tag's whole group into the #538 quarantine.
  5. Any SQL or transport failure makes the write return false, so the recorder retries and the #538 per-tag quarantine still applies. It never throws.

The recorder is Primary-only (#537), so step 2 does not race another live writer. The recorder waits for WAL visibility, bounded by VisibilityTimeout (plan R5), using the §2.3 wait.

Binding (T0-confirmed Q1/Q6/Q13):

  • Every timestamp parameter is NpgsqlDbType.TimestampTz with DateTimeKind.Utc, after Round. Timestamp + Utc throws in Npgsql. Npgsql truncates sub-µs ticks toward 2000-01-01, which rounds up before 2000. A raw DateTime.MinValue/MaxValue is sent as ±infinity, and a [Min, Max) range then silently matches nothing. R6's BindTimestamp (round; MinValue → +10 ticks; MaxValue → −9 ticks) fixes all three.

  • The other parameter types:

    Column NpgsqlDbType
    tag/str_value/other strings Varchar
    kind/data_type Smallint
    num_value Double
    int_value/status Bigint
    deleted Boolean
    severity Integer

    NULL is DBNull.Value with the column's type.

  • Every QuestDB server error carries SQLSTATE 00000. QuestDbConnection's classification therefore treats a PostgresException as a server-side rejection, judged by MessageText. Any other NpgsqlException, IOException, SocketException or timeout is transport (transient).

2.3 HistoryUpdate (soft delete)

Every operation is an INSERT, and the dedup upsert performs the replace. QuestDB UPDATE and DELETE are never issued. "Exists" means a row that is present and not deleted.

Operation Effect Result
Insert Write the row when none exists; a tombstone counts as absent GoodEntryInserted / BadEntryExists
Replace Overwrite the row when one exists GoodEntryReplaced / BadNoEntryExists
Update Write the row either way GoodEntryReplaced or GoodEntryInserted
DeleteAtTime Write a tombstone (deleted = true, value NULL) where a row exists Good / BadNoEntryExists
DeleteRaw Write a tombstone for every live row in the range the same codes SQL Server returns for the same cases

Further rules:

  • The per-item rules come from HistoryUpdateResults, shared with SQL Server, so the codes are identical.
  • Item order is preserved, with exactly one code per item.
  • Read-your-writes. After its inserts, the updater polls wal_tables() until the table's applied transaction reaches the sequencer transaction it produced, up to VisibilityTimeout. On a timeout the write is kept, the codes are reported as normal, and a Warning is logged.
  • The visibility wait (QuestDbWalVisibility, T0-confirmed Q4) is used by the writer and the updater (R5):
    1. After the INSERT(s) return: SELECT sequencerTxn FROM wal_tables() WHERE name = $1 ($1 = the full table name, Varchar) → target. Each autocommitted INSERT is exactly one sequencer transaction.
    2. Poll SELECT writerTxn, suspended FROM wal_tables() WHERE name = $1 about every 10 ms until writerTxn >= target. Stop early, with a Warning, if suspended is true. Stop at VisibilityTimeout.
    • Measured: median 0 ms, p95 6 ms, max 13 ms. Up to 7/20 inserts were not visible to an immediate re-read, so the wait is required. A concurrent writer can only raise target. The key-poll fallback (S11) is not used.
  • Existence checks for sparse item times (Insert/Replace/Update/DeleteAtTime) use SELECT ts, deleted FROM {p}sample WHERE tag = $1 AND ts IN ($2, …), chunked at 1,000 times (T0 Q11).
  • DeleteRaw (T0-confirmed), page size 1,000:
    1. SELECT ts FROM {p}sample WHERE tag = $1 AND ts >= $2 AND ts < $3 AND NOT deleted ORDER BY ts LIMIT 1000.
    2. Write one 1,000-row tombstone INSERT.
    3. Run the visibility wait.
    4. Continue from last ts + 10 ticks, until a page comes back empty. The wait between pages is required, or the next page re-reads rows just tombstoned. Measured: 2,500 rows, 3 pages, 103 ms.
  • Ruling candidate (T0 Q13): an Insert, Replace or Update item timestamped before 1970-01-01 returns BadOutOfRange, because QuestDB cannot store it. Deletes there find nothing and return BadNoEntryExists. Pre-1970 conformance seeds are skipped for QuestDB.
  • The existence checks for Insert and Replace read committed data. If a live recorder sample at the same timestamp is still in flight in the WAL, it can race the updater. Updates normally target past timestamps, so this is accepted and documented.

2.4 Retention

When RetentionDays is set, the bootstrapper applies TTL to hist_sample and hist_event, and re-applies it when the configured value changes. QuestDB then drops expired day partitions, tombstones included. When unset, data is kept forever. hist_tag never expires.

T0-confirmed (Q5):

  • Create: … PARTITION BY DAY TTL n DAYS WAL … (§2.1).
  • Change: ALTER TABLE {p}sample SET TTL n DAYS. Remove (RetentionDays unset while the table has a TTL): ALTER TABLE … SET TTL 0 DAYS.
  • n cannot be a $ parameter (invalid syntax, should be <number> <unit> but was $1). It is the validated integer, formatted with InvariantCulture.
  • TTL is not allowed on the PARTITION BY YEAR hist_tag table (TTL value must be an integer multiple of the partition size).
  • Reading the current TTL: SELECT ttlValue, ttlUnit FROM tables() WHERE table_name = $1. QuestDB normalises the unit: 7 days reads 1 WEEK, 14 days 2 WEEK, 30 days 30 DAY, none 0 HOUR. The bootstrapper compares in hours (HOUR=1, DAY=24, WEEK=168; any other unit counts as different) and issues an ALTER only on a difference.
  • ALTER … SET TTL is applied through the WAL, so tables() shows the old value for 25–100 ms. The bootstrapper does not re-read to verify.
  • Expiry is checked against the wall clock: the image default is cairo.ttl.use.wall.clock=true. A future-dated sample does not expire current data (T0 Q5c).
  • Table listing: SELECT table_name FROM tables() WHERE starts_with(table_name, $1). Never LIKE 'hist_%', because _ is a wildcard.
  • Concurrent bootstrap: CREATE TABLE IF NOT EXISTS from 4 connections × 20 rounds produced no errors. The S11 defence stays: catch a PostgresException whose MessageText contains already exists, re-check tables(), and rethrow if the table is absent. Never key on SQLSTATE (always 00000).

3. Reads

Every query filters NOT deleted. Every value goes through StoredSample, so the quality clamp is the same as SQL Server's.

  • Raw. A range on ts, ORDER BY ts [DESC], LIMIT @top (maxValuesPerNode == 0 ⇒ unbounded), at most top samples, ContinuationPoint: null; the node manager pages by re-reading from the last timestamp (plan R8). The dedup key forbids same-instant ties. Bounds follow the existing data-source contract.
    • T0-confirmed form (review fix): SELECT ts, kind, num_value, int_value, str_value, status FROM {p}sample WHERE tag = $1 AND ts >= $2 AND ts < $3 AND NOT deleted ORDER BY ts [DESC] LIMIT $4.
    • $4 is a bound parameter, NpgsqlDbType.Integer (Int32) or Bigint (Int64); both work on 10.0.1, ASC and DESC. Use Integer, since top is an int.
    • int.MaxValue (the R8 "unbounded" value) and long.MaxValue both return every row. LIMIT 0 returns no rows, but R8 never sends 0.
    • The inlined-literal fallback is not needed.
  • AtTime (T0-confirmed Q9). Find the previous-or-equal neighbour and the next neighbour of each requested time with LIMIT 1 seeks, UNION ALL-ed, at most 50 request times (100 branches) per statement. The exact QuestDB 10.0.1 limit is 126 UNION ALL branches per statement (63 request times): 126 work, and 127 fail with SQL model is too complex to evaluate. So 50 times leaves a 26-branch margin. Per time k, with $1 = tag and $j = the rounded time: 
    SELECT * FROM (SELECT k AS req, 0 AS dir, ts, kind, num_value, int_value, str_value, status FROM {p}sample
  WHERE tag = $1 AND ts <= $j AND NOT deleted ORDER BY ts DESC LIMIT 1)
UNION ALL
SELECT * FROM (SELECT k AS req, 1 AS dir, ts, kind, num_value, int_value, str_value, status FROM {p}sample
  WHERE tag = $1 AND ts > $j AND NOT deleted ORDER BY ts ASC LIMIT 1)
    k and dir are generated integer literals. Measured: 102 times in about 16–20 ms over 100k rows per tag, with no mismatches. ASOF JOIN was faster (1 ms), but it yields only the previous neighbour, so it was not chosen. AtTimeEvaluator then decides the result.
  • Events. Filtered by source and ts range, ordered ascending (reversed windows are swapped, as built today), and limited to maxEvents.
  • Processed. One query per node (T0-confirmed for every family; the S11 two-statement fallback is not needed).
    • Bucket index (T0-confirmed): (cast(ts AS LONG) * 10 - $1) / $2.
      • cast(ts AS LONG) is µs since 1970.
      • $1 = the rounded window start in ticks since the Unix epoch (start.Ticks − DateTime.UnixEpoch.Ticks).
      • $2 = the interval in ticks. Both are NpgsqlDbType.Bigint.
      • The result is exact because both sides are µs-aligned (R6), and it cannot overflow before year 9999.
    • The query returns the full BucketRow statistics. Usable, good, bad and uncertain counts come from sum(CASE …) over the status column.
    • It does not use SAMPLE BY, whose alignment rules differ from ours. Computing the index ourselves keeps the bucket edges identical to SQL Server: start inclusive, end exclusive, a partial final bucket, interval 0, and the 100k cap.
    • The time of the min or max value (ActualTime family) comes from a row_number() window function in the same single statement: ORDER BY num_value ASC|DESC, ts ASC, with ties going to the earliest sample (T0-confirmed; the form is below). No second statement is used. AggregateEvaluator then produces the values.
    • Confirmed forms (T0 Q8). They mirror SqlAggregateQueries's three families and column order. They were checked against hand-computed seeds and a 60k-row randomized cross-check across 4 partitions: 617 buckets, 0 mismatches.
      • Classified subquery s: 
        SELECT (cast(ts AS LONG) * 10 - $1) / $2 AS b, ts, kind, num_value, int_value, str_value,
       CASE WHEN status < 2147483648 AND kind <> 0 THEN 1 ELSE 0 END AS u,
       CASE WHEN status < 1073741824 AND kind <> 0 THEN 1 ELSE 0 END AS g
FROM {p}sample WHERE tag = $3 AND ts >= $4 AND ts < $5 AND NOT deleted
      • Counts (ordinals 1–5 after b): count() AS total_count, sum(u) AS usable_count, sum(g) AS good_count, count() - sum(u) AS bad_count, sum(CASE WHEN u = 1 AND kind IN (4, 5) THEN 1 ELSE 0 END) AS non_numeric_count.
      • Simple: SELECT b, <counts>, avg(CASE WHEN u = 1 THEN num_value END) AS avg_value, min(CASE WHEN u = 1 THEN num_value END) AS min_value, max(…) AS max_value, sum(…) AS sum_value FROM (<s>) GROUP BY b ORDER BY b.
      • StartEnd: 
        WITH s AS (<s>), st AS (SELECT b, <counts> FROM s GROUP BY b),
     f AS (SELECT b, kind, num_value, int_value, str_value, ts,
                  row_number() OVER (PARTITION BY b ORDER BY ts) AS rf,
                  row_number() OVER (PARTITION BY b ORDER BY ts DESC) AS rl FROM s WHERE u = 1)
SELECT st.*, x.kind, x.num_value, x.int_value, x.str_value, x.ts, y.kind, y.num_value, y.int_value, y.str_value, y.ts
FROM st LEFT JOIN (SELECT * FROM f WHERE rf = 1) x ON x.b = st.b
        LEFT JOIN (SELECT * FROM f WHERE rl = 1) y ON y.b = st.b ORDER BY st.b
      • ActualTime: the same s/st, then r AS (SELECT b, num_value, ts, row_number() OVER (PARTITION BY b ORDER BY num_value ASC|DESC, ts ASC) AS rn FROM s WHERE u = 1 AND num_value IS NOT NULL) SELECT st.*, r.num_value, r.ts FROM st LEFT JOIN (SELECT * FROM r WHERE rn = 1) r ON r.b = st.b ORDER BY st.b.
      • A bucket with no usable sample returns NULL times but kind = 0 (BYTE has no NULL). The mapper keys absence on the NULL ts.
      • first()/last() and arg_min/arg_max also returned correct values, but no ORDER BY fixes their tie order. The window forms state the earliest-timestamp tie-break explicitly and are the ones to use.
  • Errors. A failed read fails only that node, under the same data-source contract as SQL Server.

4. Configuration (ServerHistorian:QuestDb)

Key Default Validation
ConnectionString — Required when Backend=QuestDb and a consumer is enabled. An Npgsql string. T0-confirmed: Host=questdb;Port=8812;Username=admin;Password=quest;Database=qdb;ServerCompatibilityMode=NoTypeLoading (the docker-dev overlay from the host: Host=localhost;Port=18812;…). ServerCompatibilityMode=NoTypeLoading is kept: it skips the pg_type load, and every confirmed form was run with it. The string also works without it. No Reset On Close is not needed, because pooled open → close → reopen works.
TablePrefix hist_ ^[A-Za-z0-9_]+$. Only checked when a consumer is enabled.
RetentionDays unset ≥ 1 when set
CommandTimeout 00:00:30 > 0
AutoCreateSchema true —
VisibilityTimeout 00:00:05 ≥ 0. 0 disables the wait.

The Gateway keys (Endpoint, ApiKey, UseTls) are not required for QuestDb, the same as for SqlServer.

5. Testing

  • T1 refactor gate. The SQL Server integration suite (118) and unit suite pass unchanged before and after the extraction. Each moved piece of logic is falsified once against its existing test.
  • Conformance suite (tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Conformance, a class library of abstract xUnit tests). Each backend's integration project subclasses it and supplies a fixture and an IConformanceSeeder, which writes rows directly in that backend's schema.
    • It covers raw forward, reverse and paging; AtTime, including the outage, after-last and future rules; all 13 aggregates with the existing hand-computed literals; the quality contract; insert-if-absent writes and redelivery; the HistoryUpdate result codes; and over-long tag names.
    • Each legitimate backend difference is a named, documented override, never a silent skip. Microsecond rounding is the only expected one.
  • QuestDB-only tests:
    • a tombstone hides the row and a redelivery does not resurrect it;
    • the read-your-writes wait;
    • TTL is applied, verified through tables() metadata;
    • concurrent bootstrap from two instances;
    • intra-batch microsecond collisions;
    • the hist_tag upsert;
    • transient-failure behaviour, writing against a stopped port.
  • Fixture. HISTORIAN_QUESTDB_TEST_CONNECTION; when it is absent, the tests skip. Each test class uses a unique throwaway TablePrefix (t_<guid>_) and drops its tables on dispose.
  • Host. Registry and validator tests for Backend=QuestDb, beside the SqlServer ones.

6. Environment

  • QuestDB is an optional overlay file, docker-dev/docker-compose.questdb.yml, so the default rig is unchanged. It defines the questdb service:
    • image questdb/questdb:10.0.1, pinned by T0 and pushed 2026-08-24;
    • container otopcua-dev-questdb-1;
    • started with docker compose -f docker-dev/docker-compose.yml -f docker-dev/docker-compose.questdb.yml up -d --no-deps questdb.
  • It exposes PG wire on host port 18812 (→ 8812) and the web console/REST on host port 19000 (→ 9000), so it does not collide with AdminUI :9000/:9200.
  • It uses the named volume otopcua-questdb-data.
  • The central nodes gain ServerHistorian__QuestDb__ConnectionString (OTOPCUA_HISTORIAN_QUESTDB), defaulting to the questdb service.
  • No new NuGet packages.

7. Live gate (T9, run by the controller)

On docker-dev, with Backend=QuestDb:

  1. Only the Primary records, with real kinds and status, one row per value.
  2. Raw and Processed HistoryRead return identical results through the raw and UNS NodeIds, and match a direct QuestDB query.
  3. HistoryUpdate covers insert, duplicate, replace, replace-missing and delete-at-time, each confirmed by a read and by a direct query. After a delete, an outbox redelivery does not bring the row back. A WriteOperate-only key and an anonymous session are both denied.
  4. On failover, recording continues at 1×.
  5. Modbus device loss stores a BadCommunicationError Empty sample, then the BadNoCommunication outage marker, then Good on recovery.
  6. TTL is visible in table metadata when RetentionDays is set.

Evidence goes in docs/plans/2026-10-10-questdb-historian-live-gate.md, and the rig is restored afterwards.

8. Work order

Wave Task Model
W0 T0 spike. Pin the image, add the compose service, and prove every QuestDB assumption here: Npgsql parameterised multi-row inserts, DEDUP UPSERT, wal_tables() visibility, TTL syntax, microsecond timestamps over Npgsql, the min/max-time query form, AtTime neighbour lookup, concurrent CREATE TABLE IF NOT EXISTS. Amend §2–§4 with the findings before W1. controller + sonnet
W1 T1: extract Sql.Common, leaving SQL Server's behaviour unchanged opus
W2 T2 conformance suite over SQL Server ‖ T3 QuestDB skeleton (options, connection, bootstrapper, fixture) sonnet ‖ sonnet
W3 T4 writes, provisioning and alarms ‖ T5 reads and aggregates ‖ T6 updater with soft delete sonnet ‖ opus ‖ sonnet
W4 T7 provider, registry and validator, plus the conformance suite run against QuestDB sonnet
W5 T8 docs (Historian.md, Configuration.md, CLAUDE.md, docker-dev README), then a Fable whole-branch review, one fix wave, and T9 live gate sonnet / fable / opus / controller
Reference in New Issue View Git Blame Copy Permalink