fix(historian): QuestDB final-review fix wave (I-1, M-1, M-3, M-5, M-8, M-9)

- I-1: QuestDbConnection caps Npgsql Maximum Pool Size at 24 (DefaultMaxPoolSize)
  when the connection string does not set it; an explicit value is kept. The
  effective value is logged once at Information.
- M-1: QuestDbTagCatalog.EnsureAsync waits for the tag table's sequenced commits
  and re-checks before inserting, so it no longer overwrites a provisioner row
  (unit/description) that is committed but not yet applied. Integration test
  reproduces the clobber against real QuestDB (falsified).
- M-3: the pre-1970 drop Warning is rate-limited per tag (once per 10 min);
  DroppedPre1970 still counts every sample.
- M-5: QuestDbHistorianProvider is IAsyncDisposable/IDisposable and disposes the
  shared Npgsql pool; Program.cs registers the provider through a factory and
  every seam resolves it from the container, so DI disposes it at shutdown.
- M-8, M-9: docs fixes (HistoryUpdate transaction wording; folded duplicate
  HistoryUpdate paragraph). Pool-size docs updated for the new default cap.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
This commit is contained in:
Joseph Doherty
2026-10-11 03:35:35 -04:00
parent e177a50046
commit 8d82d049e9
11 changed files with 309 additions and 30 deletions
+5 -7
View File
@@ -419,19 +419,17 @@ when `ServerHistorian:Enabled` **or** `AlarmHistorian:Enabled` is true (`QuestDb
| Key | Default | Validation / meaning |
|---|---|---|
| `ServerHistorian:QuestDb:ConnectionString` | `""` | **Required.** Npgsql connection string for QuestDB's PG-wire port, e.g. `Host=questdb;Port=8812;Username=admin;Password=quest;Database=qdb;ServerCompatibilityMode=NoTypeLoading`. Env `ServerHistorian__QuestDb__ConnectionString` / `${secret:}` — never commit a real one. Set Npgsql `Maximum Pool Size` below QuestDB's 64-connection default limit (Npgsql's own default is 100), per node. |
| `ServerHistorian:QuestDb:ConnectionString` | `""` | **Required.** Npgsql connection string for QuestDB's PG-wire port, e.g. `Host=questdb;Port=8812;Username=admin;Password=quest;Database=qdb;ServerCompatibilityMode=NoTypeLoading`. Env `ServerHistorian__QuestDb__ConnectionString` / `${secret:}` — never commit a real one. When the string does not set Npgsql `Maximum Pool Size`, the backend caps it at **24** (Npgsql's own default of 100 exceeds QuestDB's 64-connection default limit); an explicit value is kept as written, and the effective value is logged at Information. Keep the per-node total of a pair under QuestDB's limit. |
| `ServerHistorian:QuestDb:TablePrefix` | `hist_` | Must match `^[A-Za-z0-9_]+$` (spliced into DDL/DML). Tables are `{prefix}sample`, `{prefix}event`, `{prefix}tag`. |
| `ServerHistorian:QuestDb:RetentionDays` | unset | `>= 1` when set. Applied as table TTL to the sample and event tables (enforced on write commits only — a quiet table never drops). Unset keeps data forever. |
| `ServerHistorian:QuestDb:CommandTimeout` | `00:00:30` | Must be `> 0`; rounded up to whole seconds. |
| `ServerHistorian:QuestDb:AutoCreateSchema` | `true` | `true` creates the tables and reconciles the TTL on first use; `false` only verifies the tables exist and fails every historian call (health `LastError`, no crash) if one is missing. |
| `ServerHistorian:QuestDb:VisibilityTimeout` | `00:00:05` | Must be `>= 0`. How long a write waits for its rows to become visible to queries (WAL apply); `0` disables the wait. A timeout only logs a Warning. |
HistoryUpdate (soft delete) needs `ServerHistorian:Enabled=true` and a `HistoryUpdate` grant, as for `SqlServer`. Semantics, the µs timestamp rules,
pre-1970 limits, TTL, and the docker-dev overlay recipe: [`docs/Historian.md`](Historian.md#questdb-backend).
HistoryUpdate (SqlServer and QuestDb only) additionally needs `ServerHistorian:Enabled=true` and a key whose principal holds the
`HistoryUpdate` permission — see [`docs/security.md`](security.md#historyupdate-historian-writes). Full semantics, schema, aggregates and the
docker-dev recipe: [`docs/Historian.md`](Historian.md#sql-server-backend).
HistoryUpdate (SqlServer and QuestDb only; QuestDb deletes are soft — tombstones) additionally needs `ServerHistorian:Enabled=true` and a key
whose principal holds the `HistoryUpdate` permission — see [`docs/security.md`](security.md#historyupdate-historian-writes). Full semantics,
schema, aggregates and the docker-dev recipe: [`docs/Historian.md`](Historian.md#sql-server-backend) (SqlServer) and
[`docs/Historian.md`](Historian.md#questdb-backend) (QuestDb: the µs timestamp rules, pre-1970 limits, TTL and the docker-dev overlay).
#### `ContinuousHistorization` (recorder + outbox knobs)
+8 -5
View File
@@ -840,9 +840,11 @@ Read only when `Backend=QuestDb`; bound by `HistorianBackendRegistry.BindQuestDb
| `QuestDb:VisibilityTimeout` | `00:00:05` | `>= 0` | How long a write waits for its rows to become visible to queries ([WAL visibility](#wal-visibility-and-the-read-your-writes-wait)). `0` disables the wait. |
⚠️ **Connection pool size.** QuestDB's default limit is **64** concurrent PG-wire connections (`pg.net.connection.limit`), while Npgsql's
default `Maximum Pool Size` is **100**. A busy node (HistoryRead batches, the recorder, the updater and alarm writes all share one pool)
can therefore be refused by QuestDB before Npgsql's own pool is full. Set `Maximum Pool Size` in the connection string below the server
limit — and below it **per node**, since a redundant pair shares one QuestDB: for example `…;Maximum Pool Size=24` on each of two nodes.
own default `Maximum Pool Size` is **100**. A busy node (HistoryRead batches, the recorder, the updater and alarm writes all share one pool)
could therefore be refused by QuestDB before Npgsql's pool is full. **When the connection string does not set `Maximum Pool Size`
(or `MaxPoolSize`), the backend now caps it at 24**, so the two nodes of a redundant pair sharing one QuestDB stay under the server limit
together (2 × 24 = 48). An explicit value in the connection string is always kept as written — including one above 64, since QuestDB's
limit is server-configurable. The effective value is logged once at Information when the backend first builds its pool.
### Tables
@@ -876,7 +878,7 @@ through `BindTimestamp`. Consequences:
timestamp by that amount (forward `T + resolution`, reverse `T − resolution`). Stepping by 1 tick would be truncated back to `T` and a
one-value-per-page forward read would loop forever. Same-timestamp ties (legal on the Gateway backend) are still paged within the timestamp.
- **Pre-1970 timestamps cannot be stored** (QuestDB: `designated timestamp before 1970-01-01 is not allowed`; the whole INSERT fails). So:
- the **live writer** drops such samples with one Warning per batch (counted in `DroppedPre1970`) — a retry could never succeed, and sending
- the **live writer** drops such samples with at most one Warning per tag per 10 minutes (a device with an unset clock would otherwise warn every drain pass; every dropped sample is still counted in `DroppedPre1970`) — a retry could never succeed, and sending
them would push the tag into the #538 quarantine;
- **HistoryUpdate** Insert / Replace / Update of such an item returns `BadOutOfRange` (nothing is sent); `DeleteAtTime` returns
`BadNoEntryExists`; `DeleteRaw` finds nothing;
@@ -1008,7 +1010,8 @@ batch limiter and `HistoryReadDeadline`.
| `DeleteRawModifiedDetails` (`IsDeleteModified=false`) | `HistoryDeleteRawModified` | one node-level code: `Good`; `BadNoData` if `[start, end)` held nothing |
| `DeleteAtTimeDetails` | `HistoryDeleteAtTime` | per time: `Good`; `BadNoEntryExists` |
Each backend call is one transaction (`MERGE … OUTPUT $action` against the TVP). The node-level `StatusCode` stays `Good` when item results are
Each backend call is applied as a unit (SqlServer: one transaction, `MERGE … OUTPUT $action` against the TVP; QuestDb: an INSERT per
chunk plus the WAL-visibility wait, no transaction — see [QuestDB backend](#questdb-backend)). The node-level `StatusCode` stays `Good` when item results are
mixed (Part 11 reports item outcomes in `OperationResults`); it is Bad only for whole-node failures.
**Per-item rejections** (answered without touching SQL; other items still proceed): a null item ⇒ `BadInvalidArgument`;