fix(historian): QuestDB final-review fix wave (I-1, M-1, M-3, M-5, M-8, M-9)
- I-1: QuestDbConnection caps Npgsql Maximum Pool Size at 24 (DefaultMaxPoolSize) when the connection string does not set it; an explicit value is kept. The effective value is logged once at Information. - M-1: QuestDbTagCatalog.EnsureAsync waits for the tag table's sequenced commits and re-checks before inserting, so it no longer overwrites a provisioner row (unit/description) that is committed but not yet applied. Integration test reproduces the clobber against real QuestDB (falsified). - M-3: the pre-1970 drop Warning is rate-limited per tag (once per 10 min); DroppedPre1970 still counts every sample. - M-5: QuestDbHistorianProvider is IAsyncDisposable/IDisposable and disposes the shared Npgsql pool; Program.cs registers the provider through a factory and every seam resolves it from the container, so DI disposes it at shutdown. - M-8, M-9: docs fixes (HistoryUpdate transaction wording; folded duplicate HistoryUpdate paragraph). Pool-size docs updated for the new default cap. Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
This commit is contained in:
@@ -419,19 +419,17 @@ when `ServerHistorian:Enabled` **or** `AlarmHistorian:Enabled` is true (`QuestDb
|
||||
|
||||
| Key | Default | Validation / meaning |
|
||||
|---|---|---|
|
||||
| `ServerHistorian:QuestDb:ConnectionString` | `""` | **Required.** Npgsql connection string for QuestDB's PG-wire port, e.g. `Host=questdb;Port=8812;Username=admin;Password=quest;Database=qdb;ServerCompatibilityMode=NoTypeLoading`. Env `ServerHistorian__QuestDb__ConnectionString` / `${secret:}` — never commit a real one. Set Npgsql `Maximum Pool Size` below QuestDB's 64-connection default limit (Npgsql's own default is 100), per node. |
|
||||
| `ServerHistorian:QuestDb:ConnectionString` | `""` | **Required.** Npgsql connection string for QuestDB's PG-wire port, e.g. `Host=questdb;Port=8812;Username=admin;Password=quest;Database=qdb;ServerCompatibilityMode=NoTypeLoading`. Env `ServerHistorian__QuestDb__ConnectionString` / `${secret:}` — never commit a real one. When the string does not set Npgsql `Maximum Pool Size`, the backend caps it at **24** (Npgsql's own default of 100 exceeds QuestDB's 64-connection default limit); an explicit value is kept as written, and the effective value is logged at Information. Keep the per-node total of a pair under QuestDB's limit. |
|
||||
| `ServerHistorian:QuestDb:TablePrefix` | `hist_` | Must match `^[A-Za-z0-9_]+$` (spliced into DDL/DML). Tables are `{prefix}sample`, `{prefix}event`, `{prefix}tag`. |
|
||||
| `ServerHistorian:QuestDb:RetentionDays` | unset | `>= 1` when set. Applied as table TTL to the sample and event tables (enforced on write commits only — a quiet table never drops). Unset keeps data forever. |
|
||||
| `ServerHistorian:QuestDb:CommandTimeout` | `00:00:30` | Must be `> 0`; rounded up to whole seconds. |
|
||||
| `ServerHistorian:QuestDb:AutoCreateSchema` | `true` | `true` creates the tables and reconciles the TTL on first use; `false` only verifies the tables exist and fails every historian call (health `LastError`, no crash) if one is missing. |
|
||||
| `ServerHistorian:QuestDb:VisibilityTimeout` | `00:00:05` | Must be `>= 0`. How long a write waits for its rows to become visible to queries (WAL apply); `0` disables the wait. A timeout only logs a Warning. |
|
||||
|
||||
HistoryUpdate (soft delete) needs `ServerHistorian:Enabled=true` and a `HistoryUpdate` grant, as for `SqlServer`. Semantics, the µs timestamp rules,
|
||||
pre-1970 limits, TTL, and the docker-dev overlay recipe: [`docs/Historian.md`](Historian.md#questdb-backend).
|
||||
|
||||
HistoryUpdate (SqlServer and QuestDb only) additionally needs `ServerHistorian:Enabled=true` and a key whose principal holds the
|
||||
`HistoryUpdate` permission — see [`docs/security.md`](security.md#historyupdate-historian-writes). Full semantics, schema, aggregates and the
|
||||
docker-dev recipe: [`docs/Historian.md`](Historian.md#sql-server-backend).
|
||||
HistoryUpdate (SqlServer and QuestDb only; QuestDb deletes are soft — tombstones) additionally needs `ServerHistorian:Enabled=true` and a key
|
||||
whose principal holds the `HistoryUpdate` permission — see [`docs/security.md`](security.md#historyupdate-historian-writes). Full semantics,
|
||||
schema, aggregates and the docker-dev recipe: [`docs/Historian.md`](Historian.md#sql-server-backend) (SqlServer) and
|
||||
[`docs/Historian.md`](Historian.md#questdb-backend) (QuestDb: the µs timestamp rules, pre-1970 limits, TTL and the docker-dev overlay).
|
||||
|
||||
#### `ContinuousHistorization` (recorder + outbox knobs)
|
||||
|
||||
|
||||
+8
-5
@@ -840,9 +840,11 @@ Read only when `Backend=QuestDb`; bound by `HistorianBackendRegistry.BindQuestDb
|
||||
| `QuestDb:VisibilityTimeout` | `00:00:05` | `>= 0` | How long a write waits for its rows to become visible to queries ([WAL visibility](#wal-visibility-and-the-read-your-writes-wait)). `0` disables the wait. |
|
||||
|
||||
⚠️ **Connection pool size.** QuestDB's default limit is **64** concurrent PG-wire connections (`pg.net.connection.limit`), while Npgsql's
|
||||
default `Maximum Pool Size` is **100**. A busy node (HistoryRead batches, the recorder, the updater and alarm writes all share one pool)
|
||||
can therefore be refused by QuestDB before Npgsql's own pool is full. Set `Maximum Pool Size` in the connection string below the server
|
||||
limit — and below it **per node**, since a redundant pair shares one QuestDB: for example `…;Maximum Pool Size=24` on each of two nodes.
|
||||
own default `Maximum Pool Size` is **100**. A busy node (HistoryRead batches, the recorder, the updater and alarm writes all share one pool)
|
||||
could therefore be refused by QuestDB before Npgsql's pool is full. **When the connection string does not set `Maximum Pool Size`
|
||||
(or `MaxPoolSize`), the backend now caps it at 24**, so the two nodes of a redundant pair sharing one QuestDB stay under the server limit
|
||||
together (2 × 24 = 48). An explicit value in the connection string is always kept as written — including one above 64, since QuestDB's
|
||||
limit is server-configurable. The effective value is logged once at Information when the backend first builds its pool.
|
||||
|
||||
### Tables
|
||||
|
||||
@@ -876,7 +878,7 @@ through `BindTimestamp`. Consequences:
|
||||
timestamp by that amount (forward `T + resolution`, reverse `T − resolution`). Stepping by 1 tick would be truncated back to `T` and a
|
||||
one-value-per-page forward read would loop forever. Same-timestamp ties (legal on the Gateway backend) are still paged within the timestamp.
|
||||
- **Pre-1970 timestamps cannot be stored** (QuestDB: `designated timestamp before 1970-01-01 is not allowed`; the whole INSERT fails). So:
|
||||
- the **live writer** drops such samples with one Warning per batch (counted in `DroppedPre1970`) — a retry could never succeed, and sending
|
||||
- the **live writer** drops such samples with at most one Warning per tag per 10 minutes (a device with an unset clock would otherwise warn every drain pass; every dropped sample is still counted in `DroppedPre1970`) — a retry could never succeed, and sending
|
||||
them would push the tag into the #538 quarantine;
|
||||
- **HistoryUpdate** Insert / Replace / Update of such an item returns `BadOutOfRange` (nothing is sent); `DeleteAtTime` returns
|
||||
`BadNoEntryExists`; `DeleteRaw` finds nothing;
|
||||
@@ -1008,7 +1010,8 @@ batch limiter and `HistoryReadDeadline`.
|
||||
| `DeleteRawModifiedDetails` (`IsDeleteModified=false`) | `HistoryDeleteRawModified` | one node-level code: `Good`; `BadNoData` if `[start, end)` held nothing |
|
||||
| `DeleteAtTimeDetails` | `HistoryDeleteAtTime` | per time: `Good`; `BadNoEntryExists` |
|
||||
|
||||
Each backend call is one transaction (`MERGE … OUTPUT $action` against the TVP). The node-level `StatusCode` stays `Good` when item results are
|
||||
Each backend call is applied as a unit (SqlServer: one transaction, `MERGE … OUTPUT $action` against the TVP; QuestDb: an INSERT per
|
||||
chunk plus the WAL-visibility wait, no transaction — see [QuestDB backend](#questdb-backend)). The node-level `StatusCode` stays `Good` when item results are
|
||||
mixed (Part 11 reports item outcomes in `OperationResults`); it is Bad only for whole-node failures.
|
||||
|
||||
**Per-item rejections** (answered without touching SQL; other items still proceed): a null item ⇒ `BadInvalidArgument`;
|
||||
|
||||
Reference in New Issue
Block a user