fix(historian): #683 review — a refused tag ADD INDEX only warns; document index trade-offs

- Bootstrapper: an ALTER ... ADD INDEX refused up front (e.g. no DDL rights)
  logs a Warning and lets the bootstrap succeed (the index is optional); a
  WAL-apply failure remains unobserved, like the TTL ALTER — documented.
  Integration test (VARCHAR tag => refused ALTER) fails with the catch removed.
- Historian.md: a dominant tag can read ~20% slower on long windows (single-
  threaded index scan vs parallel JIT filter); O3 writes rewrite index files;
  sample_modified.tag / event.source deliberately not indexed, and why.
- CLAUDE.md: the concurrent-HistoryUpdate-delete caveat on the range-first order.
This commit is contained in:
Joseph Doherty
2026-10-12 10:15:37 -04:00
parent 549d2d68b3
commit cc0cc8cd89
4 changed files with 54 additions and 7 deletions
@@ -19,7 +19,9 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb;
/// <item><b>Tag index (#683):</b> a sample table created before the <c>tag</c> symbol index existed gains it with
/// <see cref="QuestDbTables.AddTagIndex"/> (idempotent; a boot that finds it indexed issues no ALTER, and like the TTL
/// the ALTER is a WAL transaction the bootstrapper does not re-read). Building it is a one-off WAL apply — ~3 s for
/// 68M rows / 63 daily partitions measured — during which writes queue and reads keep scanning.</item>
/// 68M rows / 63 daily partitions measured — during which writes queue and reads keep scanning. A refused ALTER is
/// logged as a Warning, never fails the bootstrap (the index is optional); a failure at WAL-apply time would suspend
/// the table unseen, the same blind spot as the TTL ALTER.</item>
/// <item><b>Failure:</b> not latched. Within <see cref="BootstrapFailureCache.DefaultDuration"/> the last failure is
/// re-thrown without touching QuestDB.</item>
/// <item><c>AutoCreateSchema=false</c> only verifies the tables exist (no TTL changes) and throws
@@ -145,8 +147,21 @@ internal sealed class QuestDbSchemaBootstrapper
return;
}
// A WAL transaction: table_columns() keeps reporting false until it applies, so do NOT re-read to verify.
await using (var alter = _conn.CreateCommand(conn, QuestDbTables.AddTagIndex(table)))
// The index is optional (reads are correct without it), so a refused ALTER (e.g. no DDL rights) must not fail the
// bootstrap: it is logged and retried on the next boot. A failure at WAL-APPLY time is not seen here (QuestDB would
// suspend the table) — the same blind spot as the TTL ALTER.
try
{
await using var alter = _conn.CreateCommand(conn, QuestDbTables.AddTagIndex(table));
await alter.ExecuteNonQueryAsync(ct).ConfigureAwait(false);
}
catch (Exception ex) when (ex is not OperationCanceledException || !ct.IsCancellationRequested)
{
_log.LogWarning("Could not add the tag index to QuestDB table {Table} ({Exception}: {Message}); data-range lookups " +
"of processed reads scan partitions instead of seeking (#683). Run: {Statement}",
table, ex.GetType().Name, ex.Message, QuestDbTables.AddTagIndex(table));
return;
}
_log.LogInformation("Added the tag index to QuestDB table {Table} (#683).", table);
}