dohertj2/lmxopcua
1
0
Fork 0
Code Issues 25 Pull Requests Actions Packages Projects Releases Wiki Activity

perf(historian): index QuestDB event.source and sample_modified.tag (lmxopcua#746)

Browse Source 
Event reads filtered on source and every modified-value read filtered on tag
scanned that symbol column in every partition of the window; only
sample.tag carried an index (#683).

Both columns are now SYMBOL INDEX in the DDL, and the bootstrapper's index
migration is generalised over QuestDbTables.SymbolIndexes (sample.tag,
event.source, sample_modified.tag): a missing index is added with the same
idempotent ALTER ... ADD INDEX, Warning-only when AutoCreateSchema is false
or the ALTER is refused. Integration tests: new tables carry all three; a
pre-index table set gains them on boot and event reads answer the same.

This reverses #683's documented choice to leave the two unindexed;
docs/Historian.md carries the reasoning.

Rebased from the unpushed remed/lane-H2 branch (originally authored 2026-10-22
in session_01Ub9hGmYLQvn9XETzw7CMsT) onto current master.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
Joseph Doherty
2026-10-22 17:00:54 -04:00
parent d44e58dcd6
commit bc2f5e1889
6 changed files with 134 additions and 32 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/Historian.md
+9 -6
View File
@@ -1940,9 +1940,9 @@ makes HistoryUpdate Replace work; the writer's own insert-if-absent semantics ar
| Table | Key | Partitioning |
|---|---|---|
| `{p}sample` (`ts, tag, kind, num_value, int_value, str_value, status, deleted, server_ts`; **`tag` is a `SYMBOL INDEX`**, #683) | `(ts, tag)` | `PARTITION BY DAY` (TTL when configured) |
| `{p}event` (one-for-one with `hist.Event`) | `(ts, event_id)` | `PARTITION BY DAY` (TTL when configured) |
| `{p}event` (one-for-one with `hist.Event`; **`source` is a `SYMBOL INDEX`**) | `(ts, event_id)` | `PARTITION BY DAY` (TTL when configured) |
| `{p}tag` (`ts, name, data_type, eng_unit, description, updated_utc`) | `(ts, name)` with `ts` fixed at `1970-01-01` — an upsert by name | `PARTITION BY YEAR`, never expires |
| `{p}sample_modified` (`ts` = the sample's source time, `tag, modified_id, modification_ts, update_type, principal, api_key_id, kind, num_value, int_value, str_value, status, server_ts`) (#668) | none — append-only, no DEDUP; `modified_id` (process-monotonic ≈ UTC ticks) orders a timestamp's trail | `PARTITION BY DAY` (TTL when configured). A NULL `status` marks "no prior value" (QuestDB `BYTE` has no NULL, so `kind` is 0) |
| `{p}sample_modified` (`ts` = the sample's source time, `tag` (**`SYMBOL INDEX`**)`, modified_id, modification_ts, update_type, principal, api_key_id, kind, num_value, int_value, str_value, status, server_ts`) (#668) | none — append-only, no DEDUP; `modified_id` (process-monotonic ≈ UTC ticks) orders a timestamp's trail | `PARTITION BY DAY` (TTL when configured). A NULL `status` marks "no prior value" (QuestDB `BYTE` has no NULL, so `kind` is 0) |
**The `tag` index (#683).** Every read filters on one tag; without a symbol index QuestDB answers `tag = x` by scanning the tag
column of every partition in the time range, which made the processed-read [data-range queries](#aggregates-processed-reads) of a stopped or late-starting
@@ -1962,10 +1962,13 @@ slower on long windows**: the index scan is single-threaded where the unindexed
measurement: 4 tags / 30 M rows, 245–294 ms indexed vs 195–218 ms unindexed; with 1000 tags, 9 ms vs 13 ms — the index wins as the
tag's share of each partition shrinks).
**Deliberately not indexed:** `{p}sample_modified.tag` — read only by `IsReadModified` HistoryRead over a window, an append-only trail a
tiny fraction of the sample table's size, with no "nearest row outside the window" lookup to make it scan partitions; and
`{p}event.source` — event reads are window-bounded (`ts` range, optionally `source = x`) and never seek beyond the window, and the event table is far
smaller than the sample table. Neither has the unbounded-scan shape #683 fixes, so neither pays the index's disk and O3 cost.
**`{p}event.source` and `{p}sample_modified.tag` are indexed too** (they were deliberately left unindexed at #683, on the grounds
that both reads are window-bounded and both tables are small). A HistoryReadEvents of one notifier filters `source = x` and every
`IsReadModified` read filters `tag = x`; unindexed, each scans that column in every partition of the window — on a long window of a busy
alarm table that is the same multi-partition scan #683 measured costing seconds on the sample table. Both are created with the index now, and
the bootstrapper adds a missing one exactly as it does the sample `tag` index (`ALTER TABLE {p}event ALTER COLUMN source ADD INDEX`,
`ALTER TABLE {p}sample_modified ALTER COLUMN tag ADD INDEX`; same idempotence, same Warning-only posture with `AutoCreateSchema=false` or a
refused ALTER). The trade-off is the one measured above (disk + O3 rewrite cost), on tables far smaller than the sample table.
`status` holds the uint OPC UA status code **unsigned** in a `LONG` (so "Bad" is `status >= 2147483648`, not `< 0`); `int_value` also holds
Booleans as 0/1; `kind` is the `HistorianValueKind` in a `BYTE`. Tag names are limited to 400 characters (longer is a per-tag failure,