diff --git a/CLAUDE.md b/CLAUDE.md index cb21e377..baa899c2 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1325,7 +1325,7 @@ are no longer supported (pinned by a test). Condition teardown must forget the w ## Historian / HistoryRead -**Backend: pluggable — `ServerHistorian:Backend` = `Gateway` (default) or `SqlServer`** (see the subsection below). +**Backend: pluggable — `ServerHistorian:Backend` = `Gateway` (default), `SqlServer` or `QuestDb`** (see the subsection below). As of the gateway-integration cutover, the **Gateway** backend's historian read, alarm-write, and continuous-historization paths are all served by the **`ZB.MOM.WW.HistorianGateway`** sidecar, consumed as the Gitea-feed **`ZB.MOM.WW.HistorianGateway.Client`** gRPC package (`historian_gateway.v1`) behind a thin @@ -1340,8 +1340,9 @@ historian tagname, which is the tag's driver `FullName`. ### Pluggable backends, SQL Server and HistoryUpdate (2026-10) -- **Selection:** `ServerHistorian:Backend` (`Gateway` default | `SqlServer`; unknown fails host start). `HistorianBackendRegistry` → `IHistorianBackendProvider` builds all five seams; the gateway keys are only required for `Gateway`. `SqlServer` uses `ServerHistorian:SqlServer:{ConnectionString (required), Schema=hist, CommandTimeout=30s, AutoCreateSchema=true}`. +- **Selection:** `ServerHistorian:Backend` (`Gateway` default | `SqlServer` | `QuestDb`; unknown fails host start). `HistorianBackendRegistry` → `IHistorianBackendProvider` builds all five seams; the gateway keys are only required for `Gateway`. `SqlServer` uses `ServerHistorian:SqlServer:{ConnectionString (required), Schema=hist, CommandTimeout=30s, AutoCreateSchema=true}`. - **SQL Server** (`Driver.Historian.SqlServer`): `hist.Sample` PK `(TagId, SourceTimeUtc)` — writes are insert-if-absent, so outbox redelivery is idempotent. Stores every value kind with its **real status**; aggregates (13, simple per-sample definitions, not Part 13) are pushed down to T-SQL with bucket math in **100 ns ticks**. Docker-dev recipe, schema and aggregate semantics: `docs/Historian.md` §"SQL Server backend". Integration tests need `HISTORIAN_SQL_TEST_CONNECTION`. +- **QuestDB** (`Driver.Historian.QuestDb`, `ServerHistorian:QuestDb:*`, Npgsql over PG wire; docker-dev overlay `docker-compose.questdb.yml`): same kinds/status/aggregates/HistoryUpdate as SqlServer, soft delete by tombstone (INSERT only — never `UPDATE`/`DELETE`; every read filters `NOT deleted`). ⚠️ **Timestamps are µs** (`QuestDbConnection.Round`/`BindTimestamp`; `IHistorianDataSource.TimestampResolutionTicks` = 10 steps HistoryRead paging), **pre-1970 cannot be stored** (writer drops, HistoryUpdate `BadOutOfRange`), every write waits for WAL visibility, TTL is enforced on write commits only, set Npgsql `Maximum Pool Size` < QuestDB's 64. `docs/Historian.md` §"QuestDB backend". - **Real status fix:** the recorder used to stamp every sample Good (`192`); it now carries the driver's status. ⚠️ On the **Gateway** backend this changes nothing stored: the gateway discards quality (C-002), so AVEVA still stores Good. - **Comms loss is historized as an outage marker** (`Empty` + `BadNoCommunication`, one per tag per outage) on SqlServer; dropped (`DroppedUnsupportedKind`) on Gateway. Outbox records are **format v2** — drain or delete `ContinuousHistorization:OutboxPath` before downgrading a node. - **HistoryUpdate** (SqlServer only; Gateway ⇒ `BadHistoryOperationUnsupported`): Insert/Replace/Update/DeleteRaw/DeleteAtTime, gated in both ACL modes — flag off needs the `HistoryUpdate` role from a **Cluster-scoped** grant (+ key re-check), flag on uses `DataPlaneAuthorizer`'s HistoryUpdate operation; `WriteOperate` never implies it. The `HistoryWrite` AccessLevel bit is set only when an updater is wired at materialise time. `docs/Historian.md` §HistoryUpdate, `docs/security.md`. diff --git a/docker-dev/README.md b/docker-dev/README.md index f7706820..6dbdbcb6 100644 --- a/docker-dev/README.md +++ b/docker-dev/README.md @@ -160,6 +160,33 @@ and the script fails loudly saying so. A plain `docker compose up -d` also hides rig without the gate helping: `up -d` blocks on the `depends_on: service_healthy` conditions and returns ~30 s *after* Traefik starts, by which time the router table is long loaded. +## QuestDB historian overlay + +`docker-compose.questdb.yml` is an **optional overlay** (not part of the default rig) that adds one service, `questdb` +(`questdb/questdb:10.0.1`, container `otopcua-dev-questdb-1`, named volume `otopcua-questdb-data`), for the `ServerHistorian:Backend=QuestDb` +historian ([`docs/Historian.md`](../docs/Historian.md#questdb-backend)). Host ports: PG wire **`18812`** (→ 8812) and the web console / REST **`19000`** +(→ 9000), chosen so they do not collide with AdminUI `:9000`/`:9200`. + +```bash +# Bring up ONLY questdb (--no-deps: nothing else is touched): +docker compose -f docker-dev/docker-compose.yml -f docker-dev/docker-compose.questdb.yml up -d --no-deps questdb + +# Point the central nodes at it (the QuestDb connection string already defaults to the `questdb` service; +# override with OTOPCUA_HISTORIAN_QUESTDB). Keep the overlay -f flag on every compose command for these nodes: +OTOPCUA_HISTORIAN_ENABLED=true OTOPCUA_HISTORIAN_BACKEND=QuestDb OTOPCUA_CONTHIST_ENABLED=true \ + docker compose -f docker-dev/docker-compose.yml -f docker-dev/docker-compose.questdb.yml up -d central-1 central-2 + +# Inspect from the host (REST, no client needed): +curl -G localhost:19000/exec --data-urlencode "query=SELECT tag, count() n FROM hist_sample WHERE NOT deleted GROUP BY tag" + +# Integration tests (skipped when the variable is absent): +export HISTORIAN_QUESTDB_TEST_CONNECTION='Host=localhost;Port=18812;Username=admin;Password=quest;Database=qdb;ServerCompatibilityMode=NoTypeLoading' +dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests +``` + +Tear down with `docker compose -f docker-dev/docker-compose.yml -f docker-dev/docker-compose.questdb.yml rm -sf questdb` (add `docker volume rm otopcua-questdb-data` +to drop the data). The `admin`/`quest` credentials are QuestDB's stock defaults, not secrets. + ## Tear down ```bash diff --git a/docs/Configuration.md b/docs/Configuration.md index 54e3bb5b..597fa8cc 100644 --- a/docs/Configuration.md +++ b/docs/Configuration.md @@ -377,11 +377,13 @@ The Galaxy/MxAccess connection settings are **not an `appsettings` section.** Th > The `OTOPCUA_GALAXY_*` environment variables that v1's in-process `Galaxy.Host` consumed **no longer live in this repo** — they moved into the separately-installed mxaccessgw gateway's own config (see the v1 archive pointer in `docs/README.md` and the Galaxy overview at [`docs/drivers/Galaxy.md`](drivers/Galaxy.md)). The only Galaxy connection secret this repo touches is the gateway API key via `ApiKeySecretRef` above. -### Historian config (`ServerHistorian:Backend` — HistorianGateway or SQL Server) +### Historian config (`ServerHistorian:Backend` — HistorianGateway, SQL Server or QuestDB) The historian backend is **pluggable**, selected by `ServerHistorian:Backend` (case-insensitive; default `Gateway`; unknown value ⇒ host start fails listing the allowed values): `Gateway` is the external **`ZB.MOM.WW.HistorianGateway`** -sidecar below, `SqlServer` is a plain SQL Server database (see [`ServerHistorian:SqlServer`](#serverhistoriansqlserver-backendsqlserver)). +sidecar below, `SqlServer` is a plain SQL Server database (see [`ServerHistorian:SqlServer`](#serverhistoriansqlserver-backendsqlserver)), +and `QuestDb` is a QuestDB time-series database reached over the Postgres wire protocol (see +[`ServerHistorian:QuestDb`](#serverhistorianquestdb-backendquestdb)). The default backend is the gateway, consumed as the `ZB.MOM.WW.HistorianGateway.Client` gRPC package (the retired Wonderware TCP sidecar is documented at [`docs/drivers/Historian.Wonderware.md`](drivers/Historian.Wonderware.md)). The OtOpcUa host reads three @@ -404,13 +406,30 @@ Validated at host start (driver-role nodes) when `ServerHistorian:Enabled` **or* | Key | Default | Validation / meaning | |---|---|---| -| `ServerHistorian:Backend` | `Gateway` | `Gateway` or `SqlServer`; anything else fails host start (checked even when the historian is disabled). | +| `ServerHistorian:Backend` | `Gateway` | `Gateway`, `SqlServer` or `QuestDb`; anything else fails host start (checked even when the historian is disabled). | | `ServerHistorian:SqlServer:ConnectionString` | `""` | **Required.** Historian database connection string (not the ConfigDb). Supply via `${secret:OT-HISTORIAN-SQL}` or env `ServerHistorian__SqlServer__ConnectionString`; the database must already exist. | | `ServerHistorian:SqlServer:Schema` | `hist` | Must match `^[A-Za-z_][A-Za-z0-9_]{0,127}$` (spliced into DDL/DML). | | `ServerHistorian:SqlServer:CommandTimeout` | `00:00:30` | Must be `> 0`; rounded up to whole seconds. | | `ServerHistorian:SqlServer:AutoCreateSchema` | `true` | `true` applies the embedded schema script under an app-lock on first use; `false` only verifies `SchemaVersion >= 1` and fails every historian call (health `LastError`, no crash) if behind. | -HistoryUpdate (SqlServer only) additionally needs `ServerHistorian:Enabled=true` and a key whose principal holds the +#### `ServerHistorian:QuestDb` (`Backend=QuestDb`) + +Read only when `ServerHistorian:Backend` is `QuestDb`; the gateway keys are then not required. Validated at host start (driver-role nodes) +when `ServerHistorian:Enabled` **or** `AlarmHistorian:Enabled` is true (`QuestDbHistorianOptions.Validate()`). + +| Key | Default | Validation / meaning | +|---|---|---| +| `ServerHistorian:QuestDb:ConnectionString` | `""` | **Required.** Npgsql connection string for QuestDB's PG-wire port, e.g. `Host=questdb;Port=8812;Username=admin;Password=quest;Database=qdb;ServerCompatibilityMode=NoTypeLoading`. Env `ServerHistorian__QuestDb__ConnectionString` / `${secret:}` — never commit a real one. Set Npgsql `Maximum Pool Size` below QuestDB's 64-connection default limit (Npgsql's own default is 100), per node. | +| `ServerHistorian:QuestDb:TablePrefix` | `hist_` | Must match `^[A-Za-z0-9_]+$` (spliced into DDL/DML). Tables are `{prefix}sample`, `{prefix}event`, `{prefix}tag`. | +| `ServerHistorian:QuestDb:RetentionDays` | unset | `>= 1` when set. Applied as table TTL to the sample and event tables (enforced on write commits only — a quiet table never drops). Unset keeps data forever. | +| `ServerHistorian:QuestDb:CommandTimeout` | `00:00:30` | Must be `> 0`; rounded up to whole seconds. | +| `ServerHistorian:QuestDb:AutoCreateSchema` | `true` | `true` creates the tables and reconciles the TTL on first use; `false` only verifies the tables exist and fails every historian call (health `LastError`, no crash) if one is missing. | +| `ServerHistorian:QuestDb:VisibilityTimeout` | `00:00:05` | Must be `>= 0`. How long a write waits for its rows to become visible to queries (WAL apply); `0` disables the wait. A timeout only logs a Warning. | + +HistoryUpdate (soft delete) needs `ServerHistorian:Enabled=true` and a `HistoryUpdate` grant, as for `SqlServer`. Semantics, the µs timestamp rules, +pre-1970 limits, TTL, and the docker-dev overlay recipe: [`docs/Historian.md`](Historian.md#questdb-backend). + +HistoryUpdate (SqlServer and QuestDb only) additionally needs `ServerHistorian:Enabled=true` and a key whose principal holds the `HistoryUpdate` permission — see [`docs/security.md`](security.md#historyupdate-historian-writes). Full semantics, schema, aggregates and the docker-dev recipe: [`docs/Historian.md`](Historian.md#sql-server-backend). @@ -519,8 +538,9 @@ historian secret this repo reads from the environment is the gateway API key: | Variable | Effect / default | |---|---| | `ServerHistorian__ApiKey` | The HistorianGateway peppered-HMAC key (`histgw__`) sent as `Authorization: Bearer`. Supply via environment — **never commit**. Required when `ServerHistorian:Enabled=true` **and `Backend=Gateway`**. | -| `ServerHistorian__Backend` | `Gateway` (default) or `SqlServer`. | +| `ServerHistorian__Backend` | `Gateway` (default), `SqlServer` or `QuestDb`. | | `ServerHistorian__SqlServer__ConnectionString` | The SQL Server historian connection string. Required when `Backend=SqlServer` and the historian (or alarm history) is enabled. Supply via environment / `${secret:}` — **never commit**. | +| `ServerHistorian__QuestDb__ConnectionString` | The QuestDB historian Npgsql connection string. Required when `Backend=QuestDb` and the historian (or alarm history) is enabled. Supply via environment / `${secret:}` — **never commit**. | ### Driver integration-test / fixture sim endpoints diff --git a/docs/Historian.md b/docs/Historian.md index 2719e872..e18e8ae1 100644 --- a/docs/Historian.md +++ b/docs/Historian.md @@ -11,9 +11,10 @@ Wonderware TCP sidecar backend this replaced was retired — see [drivers/Historian.Wonderware.md](drivers/Historian.Wonderware.md).) > **The backend is pluggable (2026-10).** `ServerHistorian:Backend` selects `Gateway` (the default — everything -> described below as "the gateway") or `SqlServer` (a plain SQL Server database the server reads and writes -> itself, which also supports OPC UA **HistoryUpdate**). See [Backend selection](#backend-selection-serverhistorianbackend), -> [SQL Server backend](#sql-server-backend) and [HistoryUpdate](#historyupdate). Wherever this page says +> described below as "the gateway"), `SqlServer` (a plain SQL Server database the server reads and writes +> itself, which also supports OPC UA **HistoryUpdate**) or `QuestDb` (a QuestDB time-series database over the Postgres +> wire protocol; also supports HistoryUpdate, via soft delete). See [Backend selection](#backend-selection-serverhistorianbackend), +> [SQL Server backend](#sql-server-backend), [QuestDB backend](#questdb-backend) and [HistoryUpdate](#historyupdate). Wherever this page says > "the historian" without qualification it means the selected backend; sections that only apply to one backend say so. Design references: [docs/plans/2026-06-14-galaxy-phase-c-historian-design.md](plans/2026-06-14-galaxy-phase-c-historian-design.md) @@ -66,23 +67,23 @@ A tag that is both historized and a native alarm: (`IHistorianBackendProvider`) that builds all five historian seams — the HistoryRead data source, the continuous- historization value writer, the alarm-history writer, the tag provisioner and (optionally) the HistoryUpdate updater. `HistorianBackendRegistry` in the Host owns the name table. An unknown name fails host start with an -`OptionsValidationException` that lists the allowed values (`Gateway`, `SqlServer`), regardless of whether anything +`OptionsValidationException` that lists the allowed values (`Gateway`, `SqlServer`, `QuestDb`), regardless of whether anything consumes the historian. The existing enable gates (`ServerHistorian:Enabled`, `AlarmHistorian:Enabled`, `ContinuousHistorization:Enabled`) are unchanged — they simply get their objects from the selected provider. -| | `Gateway` (default) | `SqlServer` | -|---|---|---| -| Storage | AVEVA Historian via the HistorianGateway sidecar | `{Schema}` tables in a SQL Server database you point it at | -| Value kinds recorded | `Double`, `Int64`, `Boolean` (numeric analog only) | all: `Double`, `Int64`, `Boolean`, `String`, `DateTime`, plus `Empty` (a Bad sample with no value) | -| Status code stored | Sent as DA quality but **discarded** by the gateway (C-002); AVEVA stores every sample Good | Stored bit-for-bit (`StatusCode` int column) | -| Comms-loss / driver-Bad publishes | Dropped, counted in `DroppedUnsupportedKind` | Stored as an `Empty` outage-marker sample carrying the Bad code (see [below](#typed-values-real-status-and-outage-markers)) | -| Processed aggregates | Average, Minimum, Maximum, Total, Count | The thirteen in the [aggregate table](#aggregates-processed-reads) | -| HistoryUpdate | No (`BadHistoryOperationUnsupported`) | Yes (Insert / Replace / Update / DeleteRaw / DeleteAtTime) | -| Alarm history | gateway `SendEvent` | `Event` table (insert-if-absent) | -| Provisioning (`EnsureTags`) | numeric types only; others skipped | `MERGE` upsert of every request, all types | -| Connection | `Endpoint` + `ApiKey` + TLS keys (top-level `ServerHistorian:*`) | `ServerHistorian:SqlServer:*` | +| | `Gateway` (default) | `SqlServer` | `QuestDb` | +|---|---|---|---| +| Storage | AVEVA Historian via the HistorianGateway sidecar | `{Schema}` tables in a SQL Server database you point it at | `{prefix}sample` / `event` / `tag` WAL tables in a QuestDB instance | +| Value kinds recorded | `Double`, `Int64`, `Boolean` (numeric analog only) | all: `Double`, `Int64`, `Boolean`, `String`, `DateTime`, plus `Empty` (a Bad sample with no value) | same as `SqlServer` | +| Status code stored | Sent as DA quality but **discarded** by the gateway (C-002); AVEVA stores every sample Good | Stored bit-for-bit (`StatusCode` int column) | Stored bit-for-bit (unsigned, in a `LONG`) | +| Comms-loss / driver-Bad publishes | Dropped, counted in `DroppedUnsupportedKind` | Stored as an `Empty` outage-marker sample carrying the Bad code (see [below](#typed-values-real-status-and-outage-markers)) | Stored as an `Empty` outage-marker sample, as on `SqlServer` | +| Processed aggregates | Average, Minimum, Maximum, Total, Count | The thirteen in the [aggregate table](#aggregates-processed-reads) | The thirteen, pushed down to QuestDB SQL | +| HistoryUpdate | No (`BadHistoryOperationUnsupported`) | Yes (Insert / Replace / Update / DeleteRaw / DeleteAtTime) | Yes — soft delete (tombstones), see [below](#historyupdate-soft-delete) | +| Alarm history | gateway `SendEvent` | `Event` table (insert-if-absent) | `{prefix}event` table (dedup upsert on `(ts, event_id)`) | +| Provisioning (`EnsureTags`) | numeric types only; others skipped | `MERGE` upsert of every request, all types | Upsert into `{prefix}tag`, all types | +| Connection | `Endpoint` + `ApiKey` + TLS keys (top-level `ServerHistorian:*`) | `ServerHistorian:SqlServer:*` | `ServerHistorian:QuestDb:*` | -Recording stays **Primary-only** (#537) and **at-least-once** with the #538 per-tag quarantine on both backends; +Recording stays **Primary-only** (#537) and **at-least-once** with the #538 per-tag quarantine on every backend; HistoryRead and HistoryUpdate are served from any node (the database is shared). The historian connection string is **not** a ConfigDb string, and a driver-only node may carry it without breaking the Phase 4 "no ConfigDb" posture. @@ -114,6 +115,14 @@ and all HistoryRead calls on historized nodes return `GoodNoData` (empty, not an "Schema": "hist", "CommandTimeout": "00:00:30", "AutoCreateSchema": true + }, + "QuestDb": { + "ConnectionString": "", + "TablePrefix": "hist_", + "RetentionDays": null, + "CommandTimeout": "00:00:30", + "AutoCreateSchema": true, + "VisibilityTimeout": "00:00:05" } } } @@ -122,7 +131,7 @@ and all HistoryRead calls on historized nodes return `GoodNoData` (empty, not an | Key | Type | Default | Description | |---|---|---|---| | `Enabled` | bool | `false` | Enable the selected backend's read client. `false` → `NullHistorianDataSource` (empty reads) and no HistoryUpdate. | -| `Backend` | string | `Gateway` | `Gateway` or `SqlServer` (case-insensitive). Unknown ⇒ host start fails. See [Backend selection](#backend-selection-serverhistorianbackend). | +| `Backend` | string | `Gateway` | `Gateway`, `SqlServer` or `QuestDb` (case-insensitive). Unknown ⇒ host start fails. See [Backend selection](#backend-selection-serverhistorianbackend). | | `Endpoint` | string | `""` | **Gateway backend only.** Absolute gateway URI, e.g. `https://host:5222`. Scheme selects transport (`https://` = TLS, `http://` = h2c plaintext). Required when `Enabled`. | | `ApiKey` | string | `""` | **Gateway backend only.** The gateway peppered-HMAC key (`histgw__`) sent as `Authorization: Bearer`. Required when `Enabled`. **Supply via env `ServerHistorian__ApiKey`.** | | `UseTls` | bool | `true` | **Gateway backend only.** Connect over TLS; must match the `Endpoint` scheme. | @@ -143,6 +152,10 @@ and all HistoryRead calls on historized nodes return `GoodNoData` (empty, not an | `SqlServer:CommandTimeout` | TimeSpan | `00:00:30` | Must be `> 0` | Per-command timeout, rounded up to whole seconds (minimum 1 s). The SQL backend does **not** use `CallTimeout`. | | `SqlServer:AutoCreateSchema` | bool | `true` | — | `true`: on first use apply any missing embedded script under a `sp_getapplock` (`OtOpcUaHistorianSchema`), so concurrent nodes serialise. `false`: only verify `SchemaVersion >= 1`; if behind, **every** historian call fails with a clear error naming the missing version (surfaced in the read client's health `LastError`, never a host crash). A failed bootstrap is retried on the next call, not latched. | +**`ServerHistorian:QuestDb`** (read only when `Backend=QuestDb`; bound by `HistorianBackendRegistry.BindQuestDb`): six keys — `ConnectionString` +(required), `TablePrefix`, `RetentionDays`, `CommandTimeout`, `AutoCreateSchema`, `VisibilityTimeout`. The full table, defaults and validation +rules are in [QuestDB backend → Configuration](#configuration-serverhistorianquestdb) (and [Configuration.md](Configuration.md#serverhistorianquestdb-backendquestdb)). + **Validation rules** (`ServerHistorianOptionsValidator`, driver-role nodes only; admin-only nodes never build historian singletons): - `Backend` must name a registered backend — checked **always**, with no I/O, even when nothing is enabled. @@ -150,6 +163,9 @@ and all HistoryRead calls on historized nodes return `GoodNoData` (empty, not an (the alarm sink sources its connection from this section even when `ServerHistorian` is off). - `Backend=Gateway`: `Endpoint` must be an absolute http(s) URI, `ApiKey` non-empty, and `UseTls` must agree with the endpoint scheme. - `Backend=SqlServer`: the three `SqlServer` rules above. `Endpoint`/`ApiKey`/`UseTls` are **not** required. + - `Backend=QuestDb`: `QuestDbHistorianOptions.Validate()` (connection string required, prefix regex, `RetentionDays >= 1`, `CommandTimeout > 0`, + `VisibilityTimeout >= 0`). `Endpoint`/`ApiKey`/`UseTls` are **not** required. A malformed `TablePrefix` is also rejected by the connection helper + when the provider first builds a seam. - ⚠️ `Backend=SqlServer` with a malformed `Schema` throws `ArgumentException` while the provider is built, so it stops host start even when no consumer is enabled (the validator is consumer-gated, provider construction is not). - The alarm-only warning at startup (`AlarmHistorian` on, `ServerHistorian` off) names the selected backend. @@ -261,7 +277,7 @@ by `SqlServer`, not by `Gateway`). (#481) fans `(null, BadNoCommunication)` for every value ref of a comms-lost driver to the dependency mux, and the recorder cannot tell that from a driver's own Bad publish. So: -- **`SqlServer`:** each disconnect stores **one `Empty` / `BadNoCommunication` (`0x80310000`) sample per historized +- **`SqlServer` and `QuestDb`:** each disconnect stores **one `Empty` / `BadNoCommunication` (`0x80310000`) sample per historized tag**. A driver's own Bad publish is stored the same way (`Empty` kind plus its own code — e.g. Modbus `BadCommunicationError` `0x80050000`). This is deliberate: raw reads show the gap, and aggregates and AtTime interpolation do not bridge it. Cost: one extra row per tag per outage. @@ -288,7 +304,7 @@ rolling a node back, let the outbox drain to zero (`otopcua.historization.outbox delete the directory named by `ContinuousHistorization:OutboxPath` (accepting the loss of anything un-drained). Upgrading needs no action. -**Delivery semantics on `SqlServer`.** Writes are *insert-if-absent* on `(TagId, SourceTimeUtc)`: the outbox's +**Delivery semantics on `SqlServer`** (`QuestDb` is the same, at µs resolution and with tombstones counting as present — see [QuestDB backend](#live-writes-insert-if-absent-tombstones-included)). Writes are *insert-if-absent* on `(TagId, SourceTimeUtc)`: the outbox's at-least-once redelivery (and both pair nodes writing the same sample in a boot window) converges on one row, and the first-stored sample wins. **Duplicate timestamps inside one batch collapse to the first.** A `NaN` / `±Infinity` `Double` cannot be stored in a SQL `float`, so the **value writer stores it as an `Empty` sample** — the tag is not quarantined for @@ -797,10 +813,188 @@ dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.Integrati --- +## QuestDB backend + +Selected with `ServerHistorian:Backend=QuestDb`. Implemented in `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb` +(pure logic shared with the SQL Server backend lives in `Driver.Historian.Sql.Common`). All access is the **Postgres wire protocol via +`Npgsql`** (no ILP, no extra package). Design: [plans/2026-10-10-questdb-historian-design.md](plans/2026-10-10-questdb-historian-design.md); +the spike that pinned every QuestDB-specific form: [plans/2026-10-10-questdb-spike.md](plans/2026-10-10-questdb-spike.md). Pinned image: +`questdb/questdb:10.0.1`. + +QuestDB serves every value kind (`Double`, `Int64`, `Boolean`, `String`, `DateTime`, `Empty`), stores the real status, records comms-loss +outage markers, serves the thirteen aggregates and supports **HistoryUpdate** (soft delete), exactly like `SqlServer`. The same conformance +suite (`tests/Drivers/Historian.Conformance`) runs against both backends; the differences below are the only ones. + +### Configuration (`ServerHistorian:QuestDb`) + +Read only when `Backend=QuestDb`; bound by `HistorianBackendRegistry.BindQuestDb`, validated (consumer-gated, like `SqlServer`) by +`QuestDbHistorianOptions.Validate()`. The gateway keys (`Endpoint`, `ApiKey`, `UseTls`) are not required. + +| Key | Default | Validation | Meaning | +|---|---|---|---| +| `QuestDb:ConnectionString` | `""` | **Required** (non-blank) | Npgsql connection string for QuestDB's PG-wire port, e.g. `Host=questdb;Port=8812;Username=admin;Password=quest;Database=qdb;ServerCompatibilityMode=NoTypeLoading`. `ServerCompatibilityMode=NoTypeLoading` skips the `pg_type` load; every query form was verified with it. Supply via env `ServerHistorian__QuestDb__ConnectionString` / `${secret:}` for anything real. | +| `QuestDb:TablePrefix` | `hist_` | `^[A-Za-z0-9_]+$` | QuestDB has no schemas, so the three tables are `{prefix}sample`, `{prefix}event`, `{prefix}tag`. Spliced into DDL/DML, hence the regex. | +| `QuestDb:RetentionDays` | unset | `>= 1` when set | Applied as table TTL to `{prefix}sample` and `{prefix}event` (see [Retention](#retention-ttl)). Unset keeps data forever. | +| `QuestDb:CommandTimeout` | `00:00:30` | `> 0` | Per-command timeout, rounded up to whole seconds (minimum 1 s). | +| `QuestDb:AutoCreateSchema` | `true` | — | `true`: create the tables (and reconcile the TTL) on first use. `false`: only verify the three tables exist; if one is missing every historian call fails with `HistorianSchemaMissingException` (health `LastError`, no crash) and no TTL is touched. A failed bootstrap is not latched: it is re-thrown for 5 s without touching QuestDB, then retried. | +| `QuestDb:VisibilityTimeout` | `00:00:05` | `>= 0` | How long a write waits for its rows to become visible to queries ([WAL visibility](#wal-visibility-and-the-read-your-writes-wait)). `0` disables the wait. | + +⚠️ **Connection pool size.** QuestDB's default limit is **64** concurrent PG-wire connections (`pg.net.connection.limit`), while Npgsql's +default `Maximum Pool Size` is **100**. A busy node (HistoryRead batches, the recorder, the updater and alarm writes all share one pool) +can therefore be refused by QuestDB before Npgsql's own pool is full. Set `Maximum Pool Size` in the connection string below the server +limit — and below it **per node**, since a redundant pair shares one QuestDB: for example `…;Maximum Pool Size=24` on each of two nodes. + +### Tables + +All three are **WAL tables with `DEDUP UPSERT KEYS`**, so re-sending a row with the same key replaces it (the upsert is what makes +HistoryUpdate Replace work; the writer's own insert-if-absent semantics are built on top, below). + +| Table | Key | Partitioning | +|---|---|---| +| `{p}sample` (`ts, tag, kind, num_value, int_value, str_value, status, deleted, server_ts`) | `(ts, tag)` | `PARTITION BY DAY` (TTL when configured) | +| `{p}event` (one-for-one with `hist.Event`) | `(ts, event_id)` | `PARTITION BY DAY` (TTL when configured) | +| `{p}tag` (`ts, name, data_type, eng_unit, description, updated_utc`) | `(ts, name)` with `ts` fixed at `1970-01-01` — an upsert by name | `PARTITION BY YEAR`, never expires | + +`status` holds the uint OPC UA status code **unsigned** in a `LONG` (so "Bad" is `status >= 2147483648`, not `< 0`); `int_value` also holds +Booleans as 0/1; `kind` is the `HistorianValueKind` in a `BYTE`. Tag names are limited to 400 characters (longer is a per-tag failure, +never truncated). `{p}tag` is the **only** "is this tag known" authority (`QuestDbTagCatalog`): reads never create a tag, Insert/Update and the +live writer do, Replace and the deletes do not. + +### Timestamps: microseconds + +QuestDB's designated timestamp is a microsecond. Every inbound `DateTime` is normalised by `QuestDbConnection.Round` (Unspecified ⇒ UTC, Local +⇒ UTC, ticks truncated to a multiple of 10, `MinValue`/`MaxValue` nudged inward because Npgsql would send them as ±infinity) and bound only +through `BindTimestamp`. Consequences: + +- Two live samples for one tag **less than 1 µs apart** collapse to one; **the first wins**. A batch is deduplicated client-side after rounding. +- **Raw and Processed results carry stored (µs) timestamps.** A sub-µs request bound is rounded, and the rounded start *is* the request start, + so a Processed bucket's timestamp is the rounded window start (a sub-µs label divergence from `SqlServer`, which labels with the exact + start; values and bucket membership are identical). +- **AtTime** evaluates at the rounded time but **stamps each result with the time the client asked for**, so a client can match answers to + requests. Stored data is µs-only, so a sub-µs request adjacent to a sample is an exact hit. +- `IHistorianDataSource.TimestampResolutionTicks` is **1** by default and **10** on QuestDB. HistoryRead-Raw paging steps off a drained + timestamp by that amount (forward `T + resolution`, reverse `T − resolution`). Stepping by 1 tick would be truncated back to `T` and a + one-value-per-page forward read would loop forever. Same-timestamp ties (legal on the Gateway backend) are still paged within the timestamp. +- **Pre-1970 timestamps cannot be stored** (QuestDB: `designated timestamp before 1970-01-01 is not allowed`; the whole INSERT fails). So: + - the **live writer** drops such samples with one Warning per batch (counted in `DroppedPre1970`) — a retry could never succeed, and sending + them would push the tag into the #538 quarantine; + - **HistoryUpdate** Insert / Replace / Update of such an item returns `BadOutOfRange` (nothing is sent); `DeleteAtTime` returns + `BadNoEntryExists`; `DeleteRaw` finds nothing; + - an alarm event with such a time is a `PermanentFail` (dead-letter); + - **read bounds before 1970 work** (a Raw/Processed window may start before the epoch; bounds are never clamped, only `MinValue`/`MaxValue` nudged). +- **`Int64` `long.MinValue`.** QuestDB's `LONG` NULL is `long.MinValue`, so a stored `FromInt64(long.MinValue)` has `int_value` NULL. The read path + repairs it (`kind = Int64 ∧ int_value IS NULL ∧ num_value = −2^63` ⇒ `long.MinValue`); without the repair it would read back as `Empty`/0. Likewise + `NaN`/`±Infinity` doubles read back as NULL, so they are stored as `Empty` + `BadOutOfRange` (live) or rejected `BadInvalidArgument` (HistoryUpdate), as on `SqlServer`. + +### Live writes: insert-if-absent, tombstones included + +The recorder's writer (`QuestDbHistorianValueWriter`) has the same first-wins redelivery contract as SQL Server: + +1. group by tag, round to µs, drop in-batch duplicates (first wins), drop pre-1970 samples; +2. read the timestamps that already exist for the tag (one range query per cluster of nearby times, **`deleted` rows included**); +3. `INSERT` only the absent rows — parameterised multi-row, at most **1,000 rows per statement**; +4. [wait for WAL visibility](#wal-visibility-and-the-read-your-writes-wait). + +Because `DEDUP UPSERT` would otherwise let a re-sent sample overwrite the stored one, step 2 is what keeps outbox redelivery idempotent — and +**a live-writer redelivery never resurrects a deleted row**: a tombstone counts as present. Any SQL or transport failure returns `false` +(the recorder retries; the #538 per-tag quarantine still applies); the writer never throws. + +### WAL visibility and the read-your-writes wait + +QuestDB applies WAL commits asynchronously, so a row just inserted may not be visible to the next query (measured: up to 7 of 20 inserts invisible +to an immediate re-read). **Every write path waits** — the live writer, the HistoryUpdate updater (including between DeleteRaw pages), the tag +catalog/provisioner and the alarm writer (`QuestDbWalVisibility`): after the INSERT(s) it reads `sequencerTxn` from `wal_tables()` and polls +`writerTxn` about every 10 ms until it catches up, the table is reported `suspended` (Warning, stop), or `VisibilityTimeout` elapses (Warning). +A timeout never changes a result code and never discards data; the rows simply become visible shortly after. Measured: median 0 ms, p95 6 ms. + +### HistoryUpdate (soft delete) + +All five operations are supported, and **every one is an `INSERT`**: QuestDB `UPDATE`/`DELETE` are never issued. "Exists" means a row that is +present **and not deleted**. + +| Operation | Effect | Result | +|---|---|---| +| Insert | write the row when none exists — **a tombstone counts as absent, so Insert over a tombstone succeeds** | `GoodEntryInserted` / `BadEntryExists` | +| Replace | overwrite (dedup upsert) when a live row exists | `GoodEntryReplaced` / `BadNoEntryExists` | +| Update | write either way | `GoodEntryReplaced` or `GoodEntryInserted` | +| DeleteAtTime | write a tombstone (`deleted = true`, value NULL, `BadNoData`) where a live row exists | `Good` / `BadNoEntryExists` | +| DeleteRaw | tombstone every live row in `[start, end)`, 1,000 per page, waiting for visibility between pages | the same codes as `SqlServer` | + +The per-item codes come from the shared `HistoryUpdatePlanner`, so they equal `SqlServer`'s (including the duplicate-timestamp-in-one-request rules +above). **Every read and aggregate filters `NOT deleted`.** A client Insert at a deleted timestamp re-populates it (the OPC UA intent); only the +*live writer's* redelivery is barred from doing so. + +⚠️ **Concurrency.** The updater serializes calls **per tag, in-process only** (64 striped locks held from the existence read through the visibility +wait), so two concurrent Insert/Replace/Delete batches on one tag cannot both act on a stale read. It does **not** lock against the live writer, +nor against a second node. The existence checks read committed data, so a recorder sample at the same timestamp still in flight in the WAL can race +an Insert/Replace (updates normally target past timestamps), and two nodes writing the same *new* key concurrently can produce last-write-wins +(QuestDB has no `UPDLOCK`; the Primary gate makes it rare). This residual is accepted. + +### Reads + +Every query filters `NOT deleted`; every value goes through `StoredSample`, so the quality clamp equals `SqlServer`'s. + +- **Raw:** `ts` range, `ORDER BY ts [DESC] LIMIT $n` (bound, `int`); `ContinuationPoint` is always null — the node manager pages by re-reading + from the last timestamp ± `TimestampResolutionTicks`. +- **AtTime:** previous-or-equal and next neighbour of each time by `LIMIT 1` seeks, `UNION ALL`-ed, **50 times per statement** (QuestDB 10.0.1 refuses + more than 126 branches). `AtTimeEvaluator` decides each value, as on `SqlServer`. +- **Events:** `[start, end)` by `ts`, ordered `(ts, event_id)`, `maxEvents + 1` fetched to produce the continuation point. +- **Processed:** one query per node; bucket index `(cast(ts AS LONG) * 10 - startUnixTicks) / intervalTicks` (never `SAMPLE BY`, whose alignment + differs), so bucket edges are identical to `SqlServer` (start-inclusive, end-exclusive, partial last bucket, interval 0 ⇒ one bucket, 100,000-bucket cap). + Min/max times use `row_number()` windows with the earliest-timestamp tie-break. + +### Retention (TTL) + +`RetentionDays` becomes `TTL n DAYS` on `{p}sample` and `{p}event` (never on `{p}tag`: TTL must be a multiple of the partition size). QuestDB drops +expired day partitions itself, tombstones included. + +- At bootstrap the current TTL is read from `tables()` (`ttlValue`, `ttlUnit`). QuestDB **normalises the unit** (7 days reads `1 WEEK`), so the + bootstrapper compares in **hours** (`HOUR`=1, `DAY`=24, `WEEK`=168; any other unit counts as different) and issues + `ALTER TABLE … SET TTL n DAYS` only on a difference (`SET TTL 0 DAYS` removes it). The `ALTER` is applied through the WAL, so it does not + re-read to verify; a very large `RetentionDays` that QuestDB normalises into MONTH/YEAR is re-`ALTER`ed on every boot (harmless). +- ⚠️ **TTL is enforced only on write commits.** QuestDB checks for expired partitions when a write commits to the table, so **a quiet table never + drops anything**: partitions older than the TTL linger until the next write lands. Do not read a stale-looking partition as a broken TTL. +- Expiry is judged against the wall clock, so a future-dated sample does not expire current data. +- The TTL is visible in table metadata: `SELECT table_name, ttlValue, ttlUnit FROM tables() WHERE starts_with(table_name, 'hist_')`. + +### Docker-dev recipe + +QuestDB is an **optional compose overlay**, `docker-dev/docker-compose.questdb.yml`, so the default rig is unchanged. See +[`docker-dev/README.md`](../docker-dev/README.md#questdb-historian-overlay). + +```bash +# 1. Bring up ONLY the questdb service (PG wire on host :18812, console/REST on host :19000). +docker compose -f docker-dev/docker-compose.yml -f docker-dev/docker-compose.questdb.yml up -d --no-deps questdb + +# 2. Recreate the central nodes against it (the QuestDb connection string already defaults to the `questdb` service): +OTOPCUA_HISTORIAN_ENABLED=true OTOPCUA_HISTORIAN_BACKEND=QuestDb OTOPCUA_CONTHIST_ENABLED=true \ + docker compose -f docker-dev/docker-compose.yml -f docker-dev/docker-compose.questdb.yml up -d central-1 central-2 + +# 3. Mark a rig tag historized, deploy, then watch rows land (REST, no client needed): +curl -G localhost:19000/exec --data-urlencode "query=SELECT tag, count() n, max(ts) last FROM hist_sample WHERE NOT deleted GROUP BY tag" +``` + +Only the Primary of a pair records. Read back with `Client.CLI historyread`, rewrite with `Client.CLI historyupdate`. + +### Integration tests + +`tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests` (bootstrap incl. two concurrent bootstrappers, TTL via `tables()`, writes, +tombstones and redelivery, the visibility wait, updater, reads, paging, and the shared conformance suite) is gated on +**`HISTORIAN_QUESTDB_TEST_CONNECTION`** — an Npgsql connection string for the PG-wire port. Absent ⇒ every test skips. Each test class uses a +unique throwaway prefix (`t__`) and drops its tables afterwards. The QuestDB server allows 64 connections, so the fixture prunes idle pool connections. + +```bash +docker compose -f docker-dev/docker-compose.yml -f docker-dev/docker-compose.questdb.yml up -d --no-deps questdb +export HISTORIAN_QUESTDB_TEST_CONNECTION='Host=localhost;Port=18812;Username=admin;Password=quest;Database=qdb;ServerCompatibilityMode=NoTypeLoading' +dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests +``` + +--- + ## HistoryUpdate OPC UA **HistoryUpdate** lets a client insert, replace, update and delete stored raw history. It is available only when the selected -backend provides an updater — **`SqlServer` does; `Gateway` does not**, and an updater is registered only when `ServerHistorian:Enabled=true`. +backend provides an updater — **`SqlServer` and `QuestDb` do; `Gateway` does not**, and an updater is registered only when `ServerHistorian:Enabled=true`. Updates go straight to the backend (not through the outbox), are **not Primary-gated** (the database is shared), and share the HistoryRead batch limiter and `HistoryReadDeadline`. diff --git a/docs/plans/2026-10-10-questdb-historian-design.md b/docs/plans/2026-10-10-questdb-historian-design.md index c2f1ec99..2e6a3ba2 100644 --- a/docs/plans/2026-10-10-questdb-historian-design.md +++ b/docs/plans/2026-10-10-questdb-historian-design.md @@ -1,6 +1,7 @@ # QuestDB historian backend — design -**Status:** approved section by section on 2026-10-10. Gitea #666. +**Status:** approved section by section on 2026-10-10. Gitea #666. **As-built pass (T8, 2026-10-11):** every section below was re-checked against +the merged code; where the two disagreed the text was corrected inline and marked **(as built)**. User documentation: `docs/Historian.md` §"QuestDB backend". **Builds on:** `docs/plans/2026-10-09-pluggable-historian-design.md`, which defines the provider seam, typed values, the HistoryUpdate service and the SQL Server backend, all merged to master as `8ec7a83c`. @@ -12,7 +13,7 @@ HistoryUpdate service and the SQL Server backend, all merged to master as `8ec7a | Write path | **Postgres wire protocol only, via `Npgsql`.** It is already centrally pinned at 10.0.3. No ILP, no new NuGet package. | | Retention | **Included.** `RetentionDays` is applied as table TTL. QuestDB drops expired day partitions itself. | | Approach | **B: shared dialect-free core plus per-backend SQL.** The pure logic moves out of the SQL Server backend into `Driver.Historian.Sql.Common`. One conformance suite runs against both backends. | -| Timestamps | **Microseconds.** Postgres-wire timestamps are microsecond resolution, and real source timestamps are at best milliseconds. Two samples less than 1 µs apart within a batch collapse to one, and the first wins. This is documented. | +| Timestamps | **Microseconds.** Postgres-wire timestamps are microsecond resolution, and real source timestamps are at best milliseconds. Two samples less than 1 µs apart collapse to one, and the first wins — within a batch, and also against an already-stored row (the existence check runs on the rounded time). This is documented. | Out of scope: - event HistoryUpdate; @@ -62,8 +63,12 @@ seam classes, all calling the shared core. | `QuestDbConnection` | Npgsql `NpgsqlDataSource` (pooled), command timeouts, transient-error classification, UTC timestamp reads. | | `QuestDbSchemaBootstrapper` | Idempotent `CREATE TABLE IF NOT EXISTS` for the three tables, then applies TTL. It is concurrency-safe across two nodes starting together, and caches a failure for 5 s. | | `QuestDbHistorianValueWriter` | `IHistorianValueWriter` for live samples (§2.2). | -| `QuestDbTagProvisioner` | `IHistorianProvisioning`: upserts `hist_tag` rows. | -| `QuestDbAlarmHistorianWriter` | `IAlarmHistorianWriter`: writes events with the deterministic SHA-256 `event_id` and the same classify rules as SQL Server (unknown ⇒ RetryPlease). | +| `QuestDbTagCatalog` | **(as built)** The only "is this tag known" authority (R7): a `hist_tag` row, positive results cached. Reads never create a tag; the live writer and HistoryUpdate Insert/Update do (Replace and the deletes do not). Re-inserting a known tag is skipped, because the dedup upsert would erase the unit and description the provisioner wrote. | +| `QuestDbWalVisibility` | **(as built)** The read-your-writes wait (§2.3), used by every write path. | +| `QuestDbTimeClusters` | **(as built)** Groups request times into ranges (gap > 1 min starts a new one) so existence reads are one range query per cluster. | +| `QuestDbTables`, `QuestDbAggregateQueries` | **(as built)** DDL / TTL helpers, and the push-down aggregate SQL (§3). | +| `QuestDbTagProvisioner` | `IHistorianProvisioning`: upserts `hist_tag` rows in chunks; **(as built)** a request without a unit or description keeps the stored one. | +| `QuestDbAlarmHistorianWriter` | `IAlarmHistorianWriter`: **(as built)** one INSERT per event (not batched), with the deterministic SHA-256 `event_id`. Outcomes: Ack; `RetryPlease` for transient and unclassified errors; `PermanentFail` only for a pre-1970 time or a server message naming an inconvertible / out-of-range / too-long / null value. | | `QuestDbHistorianDataSource` | `IHistorianDataSource` (§3), with QuestDB SQL for raw, at-time, events and bucket statistics. | | `QuestDbHistorianDataUpdater` | `IHistorianDataUpdater` (§2.3), the soft-delete updater. | | `QuestDbHistorianProvider` | `IHistorianBackendProvider`: `Name = "QuestDb"`, `SupportedKinds` = all six (FrozenSet), one shared connection and bootstrapper. | @@ -73,6 +78,8 @@ Host changes: - `ServerHistorianOptionsValidator` validates the QuestDb section, gated on consumers, the same way it gates SqlServer. - `Host.csproj` gets a ProjectReference. +- **(as built)** `IHistorianDataSource` gains the default member `TimestampResolutionTicks` (1; QuestDB 10), and the node manager's Raw paging steps off a drained timestamp by it (§3). +- **(as built)** The provider builds its shared pieces (connection, bootstrapper, catalog, WAL wait) on the first `Create*` call, not in the constructor, so a provider nobody consumes never fails on its options. ## 2. Schema and writes @@ -132,11 +139,11 @@ Column details (**T0-confirmed**): - `long.MinValue` (LONG) and `int.MinValue` (INT) are QuestDB's NULL sentinels. - `BYTE` and `BOOLEAN` have **no NULL**: unset or DBNull reads `0` / `false`. So `deleted` defaults to `false`, and a LEFT-JOINed "no row" yields `kind = 0`. Absence must be detected on a NULL `ts`, not on `kind`. -- **Ruling candidate (T0):** an Int64 sample of exactly `long.MinValue` stores `int_value` as NULL. The QuestDB read +- **Ruling RC2 (as built):** an Int64 sample of exactly `long.MinValue` stores `int_value` as NULL. The QuestDB read path repairs it before the shared `FromColumns`: `kind = Int64 ∧ int_value IS NULL ∧ num_value = -2^63` ⇒ `long.MinValue`. That double is exact. Without the repair the sample would be served as 0. - A designated timestamp **before 1970-01-01 cannot be inserted** (`designated timestamp before 1970-01-01 is not - allowed`). See §2.2. + allowed`). See §2.2 (RC1). - Tag names: SYMBOL and VARCHAR take 400 characters (non-ASCII included). SYMBOL grows past its capacity hint (16,384) with no error, and was tested with 30,000 distinct tags. No `CAPACITY` clause is needed. @@ -144,20 +151,24 @@ Column details (**T0-confirmed**): 1. Group the batch by tag and remove duplicates within the batch. **The first value wins.** A timestamp is rounded to microseconds before this step, so microsecond collisions also collapse. -2. For each tag, fetch the timestamps that already exist in the batch's `[min ts, max ts]` range. **Tombstones are - included**, so a re-sent outbox batch can never resurrect a deleted row or overwrite a HistoryUpdate change. - T0-confirmed form (Q11): `SELECT ts FROM {p}sample WHERE tag = $1 AND ts >= $2 AND ts <= $3`, with no `NOT - deleted`, followed by a client-side set test. +2. For each tag, fetch the timestamps that already exist. **(as built)** The batch's times are grouped into clusters + (`QuestDbTimeClusters`: a gap over 1 minute starts a new range; the live writer also caps a range at 1,000 times), with one range query per cluster, so one + far-away outlier does not widen a single query over every stored row in between. **Tombstones are included**, so a re-sent + outbox batch can never resurrect a deleted row or overwrite a HistoryUpdate change. T0-confirmed form (Q11): `SELECT ts + FROM {p}sample WHERE tag = $1 AND ts >= $2 AND ts <= $3`, with no `NOT deleted`, followed by a client-side set test. The tag is + registered in `QuestDbTagCatalog` first (a write creates an unknown tag). 3. Insert only the new rows with parameterised multi-row `INSERT … VALUES (…),(…)`, **at most 1,000 rows per statement** (T0-confirmed Q2). QuestDB reads the Bind parameter count as a signed int16, so the hard limit is - 32,767 parameters (3,640 nine-column rows). Within one statement a duplicate key keeps the **last** row, which is + 32,767 parameters. **(as built)** The live writer binds 8 parameters per row (`deleted` is the literal `false`; 4,095 rows + would fit), the updater 9 (3,640 rows). Within one statement a duplicate key keeps the **last** row, which is why step 1's "first wins" is done client-side. 4. Stored-value rules are shared with SQL Server: - NaN/±Inf, or any Empty with a non-Bad status, is stored Empty with `BadOutOfRange` or `BadNoData`. - Comms-loss outage markers are stored as given. - A tag name over 400 characters is a per-tag failure. - - **Ruling candidate (T0 Q13):** a sample timestamped before 1970-01-01 cannot be stored, because QuestDB rejects - the whole INSERT. The writer drops such samples before the INSERT and logs one Warning per batch. A retry could + - **Ruling RC1 (T0 Q13):** a sample timestamped before 1970-01-01 cannot be stored, because QuestDB rejects + the whole INSERT. The writer drops such samples before the INSERT, logs one Warning per batch and counts them + (`DroppedPre1970`); the call still succeeds for the rest. A retry could never succeed, and sending the sample would push the tag's whole group into the #538 quarantine. 5. Any SQL or transport failure makes the write return `false`, so the recorder retries and the #538 per-tag quarantine still applies. It never throws. @@ -166,7 +177,10 @@ The recorder is Primary-only (#537), so step 2 does not race another live writer visibility, bounded by `VisibilityTimeout` (plan R5), using the §2.3 wait. Binding (T0-confirmed Q1/Q6/Q13): -- Every timestamp parameter is `NpgsqlDbType.TimestampTz` with `DateTimeKind.Utc`, after `Round`. `Timestamp` + `Utc` +- Every timestamp parameter is `NpgsqlDbType.TimestampTz` with `DateTimeKind.Utc`, after `Round` (**as built:** Unspecified is + taken as UTC, Local is converted, then ticks are truncated to a multiple of 10). `BindTimestamp(…, clampToEpoch: true)` is used + only for write binds of a designated timestamp; read bounds are never clamped (a pre-1970 bound is valid, and clamping it would + move the bound across the epoch row). `Timestamp` + `Utc` throws in Npgsql. Npgsql truncates sub-µs ticks **toward 2000-01-01**, which rounds *up* before 2000. A raw `DateTime.MinValue`/`MaxValue` is sent as ±infinity, and a `[Min, Max)` range then silently matches nothing. R6's `BindTimestamp` (round; `MinValue` → `+10` ticks; `MaxValue` → `−9` ticks) fixes all three. @@ -193,7 +207,7 @@ issued. "Exists" means a row that is present **and not deleted**. | Operation | Effect | Result | |---|---|---| -| Insert | Write the row when none exists; a tombstone counts as absent | `GoodEntryInserted` / `BadEntryExists` | +| Insert | Write the row when none exists; **a tombstone counts as absent, so Insert over a tombstone returns `GoodEntryInserted`** (an explicit client insert may re-populate a deleted time; only the live writer's redelivery is barred, §2.2) | `GoodEntryInserted` / `BadEntryExists` | | Replace | Overwrite the row when one exists | `GoodEntryReplaced` / `BadNoEntryExists` | | Update | Write the row either way | `GoodEntryReplaced` or `GoodEntryInserted` | | DeleteAtTime | Write a tombstone (`deleted = true`, value NULL) where a row exists | `Good` / `BadNoEntryExists` | @@ -212,8 +226,12 @@ Further rules: target`. Stop early, with a Warning, if `suspended` is true. Stop at `VisibilityTimeout`. - Measured: median 0 ms, p95 6 ms, max 13 ms. Up to 7/20 inserts were not visible to an immediate re-read, so the wait is required. A concurrent writer can only raise `target`. The key-poll fallback (S11) is not used. -- Existence checks for sparse item times (Insert/Replace/Update/DeleteAtTime) use `SELECT ts, deleted FROM {p}sample - WHERE tag = $1 AND ts IN ($2, …)`, chunked at 1,000 times (T0 Q11). +- **(as built)** Existence checks for item times (Insert/Replace/Update/DeleteAtTime) are **not** `IN` lists: they use the T0 Q11 range + form, one query per `QuestDbTimeClusters` cluster, `SELECT ts FROM {p}sample WHERE tag = $1 AND ts >= $2 AND ts <= $3 AND NOT + deleted`, and the set test is client-side. The `IN ($2, …)` form this section originally proposed is not used. +- **Unknown tag (as built):** Insert and Update register the tag in `QuestDbTagCatalog`; Replace and the deletes do not (Replace ⇒ + `BadNoEntryExists`, DeleteAtTime ⇒ `BadNoEntryExists`, DeleteRaw ⇒ `BadNoData`). +- **Tombstone shape (as built):** `kind = Empty (0)`, value columns NULL, `status = BadNoData`, `deleted = true`. - **DeleteRaw (T0-confirmed), page size 1,000:** 1. `SELECT ts FROM {p}sample WHERE tag = $1 AND ts >= $2 AND ts < $3 AND NOT deleted ORDER BY ts LIMIT 1000`. 2. Write one 1,000-row tombstone INSERT. @@ -221,12 +239,17 @@ Further rules: 4. Continue from `last ts + 10 ticks`, until a page comes back empty. The wait between pages is required, or the next page re-reads rows just tombstoned. Measured: 2,500 rows, 3 pages, 103 ms. -- **Ruling candidate (T0 Q13):** an Insert, Replace or Update item timestamped before 1970-01-01 returns - `BadOutOfRange`, because QuestDB cannot store it. Deletes there find nothing and return `BadNoEntryExists`. Pre-1970 - conformance seeds are skipped for QuestDB. -- The existence checks for Insert and Replace read committed data. If a live recorder sample at the *same* - timestamp is still in flight in the WAL, it can race the updater. Updates normally target past timestamps, so this - is accepted and documented. +- **Ruling RC1 (T0 Q13):** an Insert, Replace or Update item timestamped before 1970-01-01 returns + `BadOutOfRange`, because QuestDB cannot store it (nothing is sent; a null or `MinValue` timestamp is still + `BadInvalidTimestamp`, and **as built** a pre-1970 item that is also NaN reports `BadOutOfRange`, not `BadInvalidArgument`, + because the pre-1970 filter runs first). DeleteAtTime there returns `BadNoEntryExists`; DeleteRaw finds nothing. A DeleteRaw + *window* may start before 1970. Pre-1970 conformance seeds are skipped for QuestDB. +- **Concurrency (as built).** The updater serializes calls **per tag, in-process only** (64 striped async locks, held from the + existence read through the visibility wait), so two concurrent Insert/Replace/Delete batches on one tag cannot both act on a stale + read. It does **not** lock against the live writer or a second node. The existence checks read committed data, so a live recorder + sample at the *same* timestamp still in flight in the WAL can race the updater, and **two nodes writing the same new key + concurrently can produce last-write-wins** (QuestDB has no `UPDLOCK`; the Primary gate on the recorder makes it rare). Updates + normally target past timestamps, so this residual is accepted and documented. ### 2.4 Retention @@ -242,10 +265,13 @@ T0-confirmed (Q5): formatted with `InvariantCulture`. - TTL is not allowed on the `PARTITION BY YEAR` `hist_tag` table (`TTL value must be an integer multiple of the partition size`). +- **TTL is enforced only on write commits (as built, N3).** QuestDB looks for expired partitions when a write commits to the table, + so a quiet table never drops anything; old partitions linger until the next write. - **Reading the current TTL:** `SELECT ttlValue, ttlUnit FROM tables() WHERE table_name = $1`. QuestDB **normalises the unit**: 7 days reads `1 WEEK`, 14 days `2 WEEK`, 30 days `30 DAY`, none `0 HOUR`. The bootstrapper compares in - hours (`HOUR`=1, `DAY`=24, `WEEK`=168; any other unit counts as different) and issues an `ALTER` only on a - difference. + hours (`HOUR`=1, `DAY`=24, `WEEK`=168; any other unit counts as different, RC3) and issues an `ALTER` only on a + difference. A large `RetentionDays` that QuestDB normalises to MONTH/YEAR therefore re-`ALTER`s on every boot (harmless). + `AutoCreateSchema=false` touches no TTL. - `ALTER … SET TTL` is applied through the WAL, so `tables()` shows the old value for 25–100 ms. The bootstrapper does not re-read to verify. - Expiry is checked against the **wall clock**: the image default is `cairo.ttl.use.wall.clock=true`. A future-dated @@ -261,7 +287,11 @@ T0-confirmed (Q5): Every query filters `NOT deleted`. Every value goes through `StoredSample`, so the quality clamp is the same as SQL Server's. -- **Raw.** A range on `ts`, `ORDER BY ts [DESC]`, `LIMIT @top` (`maxValuesPerNode == 0` ⇒ unbounded), at most `top` samples, `ContinuationPoint: null`; the node manager pages by re-reading from the last timestamp (plan R8). The dedup key forbids same-instant ties. Bounds follow the existing data-source contract. +- **Raw.** A range on `ts`, `ORDER BY ts [DESC]`, `LIMIT @top` (`maxValuesPerNode == 0` ⇒ unbounded), at most `top` samples, `ContinuationPoint: null`; the node manager pages by re-reading from the last timestamp (plan R8). The dedup key forbids same-instant ties. Bounds follow the existing data-source contract: forward `[start, end)`, reverse `(end, start]` + descending, and **(as built)** `start == end` reads the sample exactly at that (rounded) time. **(as built)** The data source reports + `TimestampResolutionTicks = 10`, and the node manager steps off the last drained timestamp by that amount (forward `T + 10`, reverse + `T − 10`): a 1-tick step is truncated back to `T`, so a one-value-per-page forward read would loop forever. The rule is *not* "trim + everything ≤ the last emitted", because the same paging serves the Gateway backend, where same-timestamp ties are legal. - **T0-confirmed form (review fix):** `SELECT ts, kind, num_value, int_value, str_value, status FROM {p}sample WHERE tag = $1 AND ts >= $2 AND ts < $3 AND NOT deleted ORDER BY ts [DESC] LIMIT $4`. - `$4` is a **bound** parameter, `NpgsqlDbType.Integer` (Int32) or `Bigint` (Int64); both work on 10.0.1, ASC and @@ -283,15 +313,19 @@ Server's. ``` `k` and `dir` are generated integer literals. Measured: 102 times in about 16–20 ms over 100k rows per tag, with no mismatches. `ASOF JOIN` was faster (1 ms), but it yields only the previous neighbour, so it was not chosen. - `AtTimeEvaluator` then decides the result. -- **Events.** Filtered by source and `ts` range, ordered ascending (reversed windows are swapped, as built today), - and limited to `maxEvents`. + `AtTimeEvaluator` then decides the result. **(as built, M3)** Each time is evaluated at its *rounded* value but the result is stamped + with the time the client *requested* (UTC-normalised, not truncated), as on SQL Server, so answers match requests. Stored data is µs-only, + so a sub-µs request adjacent to a sample is an exact hit. +- **Events.** Filtered by source and `ts` range `[start, end)`, ordered `(ts, event_id)` ascending (reversed windows are swapped, as built + today). **(as built)** With a bound, `maxEvents + 1` rows are fetched; the extra row is dropped and becomes the continuation point. - **Processed.** **One query per node** (T0-confirmed for every family; the S11 two-statement fallback is not needed). - Bucket index (T0-confirmed): `(cast(ts AS LONG) * 10 - $1) / $2`. - `cast(ts AS LONG)` is µs since 1970. - `$1` = the rounded window start in **ticks since the Unix epoch** (`start.Ticks − DateTime.UnixEpoch.Ticks`). - `$2` = the interval in ticks. Both are `NpgsqlDbType.Bigint`. - The result is exact because both sides are µs-aligned (R6), and it cannot overflow before year 9999. + - **(as built)** The bucket timestamp returned for a Processed bucket is the *rounded* window start plus `i × interval`, so with a + sub-µs request start it differs from SQL Server's label by the sub-µs remainder; values and membership are identical. - The query returns the full `BucketRow` statistics. Usable, good, bad and uncertain counts come from `sum(CASE …)` over the status column. - It does **not** use `SAMPLE BY`, whose alignment rules differ from ours. Computing the index ourselves keeps the @@ -345,6 +379,11 @@ Server's. The Gateway keys (`Endpoint`, `ApiKey`, `UseTls`) are not required for QuestDb, the same as for SqlServer. +**(as built) Pool size.** QuestDB's default limit is **64** PG-wire connections; Npgsql's default `Maximum Pool Size` is **100**, and the +read, write, update and alarm paths share one pool per node. Operators should set `Maximum Pool Size` in the connection string below the +server limit, per node (a redundant pair shares one QuestDB). The validator checks the prefix only when a consumer is enabled, but the +connection helper also refuses a malformed prefix (`ArgumentException`) when the first seam is built. + ## 5. Testing - **T1 refactor gate.** The SQL Server integration suite (118) and unit suite pass unchanged before and after the @@ -364,7 +403,11 @@ The Gateway keys (`Endpoint`, `ApiKey`, `UseTls`) are not required for QuestDb, - concurrent bootstrap from two instances; - intra-batch microsecond collisions; - the `hist_tag` upsert; - - transient-failure behaviour, writing against a stopped port. + - transient-failure behaviour, writing against a stopped port; + - **(as built)** pre-1970 handling (writer drop, `BadOutOfRange`, read bounds), the `long.MinValue` read repair, Insert over a tombstone, + per-tag updater serialization, and a forward cap-1 paging regression against a real QuestDB. + - Conformance differences are named overrides: pre-1970 seeds are skipped on QuestDB, and AtTime timestamp assertions compare `Round(x)` to + the requested time (identity on SQL Server). The QuestDB fixture prunes idle pool connections after 1 s because of the 64-connection limit. - **Fixture.** `HISTORIAN_QUESTDB_TEST_CONNECTION`; when it is absent, the tests skip. Each test class uses a unique throwaway `TablePrefix` (`t__`) and drops its tables on dispose. - **Host.** Registry and validator tests for `Backend=QuestDb`, beside the SqlServer ones.