dohertj2/lmxopcua
1
0
Fork 0
Code Issues 24 Pull Requests Actions 2 Packages Projects Releases Wiki Activity

fix(historian): QuestDB modification trail gets an applied marker and a dedup key (lmxopcua#746)

Browse Source 
QuestDB has no multi-table transaction, so a HistoryUpdate writes its trail
row before the sample change. A failed or crashed change left an orphan
trail row that modified reads served as a change that happened, and a trail
INSERT re-sent after a client timeout duplicated (the table had no key).

- sample_modified gains `pending` and DEDUP UPSERT KEYS(ts, tag,
  modified_id). The trail row is written pending; after the change lands
  the same row is re-upserted pending = false (the applied marker).
- A transient trail INSERT failure is re-sent once with the same ids; a
  first attempt that did apply converges on the key.
- ReadModifiedAsync serves a pending row only when it is the latest trail
  row at its time and the sample there was written at or after the
  modification (a crash between change and marker). The orphan of a change
  that never landed, typically followed by the client's retry, is dropped.
  ASOF JOIN over the window's samples of one tag; the resume filter sits
  outside the window function so it never cuts a time's partition.
- The bootstrapper migrates an older trail (ADD COLUMN pending, DEDUP
  ENABLE) and waits for it to apply (bounded 10 s); old rows read
  pending = false and stay visible. Until applied, or with
  AutoCreateSchema=false, the old format is used.

Rebased from the unpushed remed/lane-H2 branch (originally authored 2026-10-22
in session_01Ub9hGmYLQvn9XETzw7CMsT) onto current master.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
Joseph Doherty
2026-10-22 17:20:19 -04:00
parent 9c3c9b0d61
commit 1a7a8f6f22
8 changed files with 462 additions and 80 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/Historian.md
+19 -11
View File
@@ -79,7 +79,7 @@ consumes the historian. The existing enable gates (`ServerHistorian:Enabled`, `A
| Comms-loss / driver-Bad publishes | Dropped, counted in `DroppedUnsupportedKind` | Stored as an `Empty` outage-marker sample carrying the Bad code (see [below](#typed-values-real-status-and-outage-markers)) | Stored as an `Empty` outage-marker sample, as on `SqlServer` |
| Processed aggregates | Twelve: the ten per-sample ones (Average, Count, Minimum, Maximum, Range, Start, End, Delta, Min/MaxActualTime) evaluated from raw reads with the SQL backends' evaluator — identical results (#681) — plus TimeAverage and Total from AVEVA's native modes | The seventeen in the [aggregate table](#aggregates-processed-reads) (incl. Part 13 time-weighted, #671) | The same seventeen, identical results |
| HistoryUpdate | No (`BadHistoryOperationUnsupported`) | Yes (Insert / Replace / Update / DeleteRaw / DeleteAtTime) | Yes — soft delete (tombstones), see [below](#historyupdate-soft-delete) |
| Modified-value history (`IsReadModified`, #668) | No (`BadHistoryOperationUnsupported`) | Yes — `SampleModified`, atomic with the change | Yes — `{prefix}sample_modified`, written before the change (orphan-on-crash) |
| Modified-value history (`IsReadModified`, #668) | No (`BadHistoryOperationUnsupported`) | Yes — `SampleModified`, atomic with the change | Yes — `{prefix}sample_modified`, written before the change, marked applied after it (an orphan is not served) |
| Alarm history | gateway `SendEvent`, one RPC per event, up to 8 in flight (no batch RPC yet) | `Event` table (insert-if-absent), multi-row statements of 150 events | `{prefix}event` table (dedup upsert on `(ts, event_id)`), multi-row INSERTs of 500 events |
| Provisioning (`EnsureTags`) | numeric types only; others skipped | `MERGE` upsert of every request, all types | Upsert into `{prefix}tag`, all types |
| Connection | `Endpoint` + `ApiKey` + TLS keys (top-level `ServerHistorian:*`) | `ServerHistorian:SqlServer:*` | `ServerHistorian:QuestDb:*` |
@@ -1094,9 +1094,12 @@ modification time and the Part 11 `HistoryUpdateType`. A HistoryReadRaw with `Is
- **Atomicity differs by backend.** `SqlServer` writes the trail row with `OUTPUT … INTO` in the **same statement** that changes `hist.Sample`,
so a change and its trail row commit or roll back together. QuestDB has no multi-table transaction, so the trail row is written **before**
the change: a crash (or failed sample INSERT) in between leaves an **orphan trail row** describing a change that never landed — never a
change without its trail row. Treat a QuestDB trail entry whose sample still shows the prior value as such an orphan.
A failed sample write after its trail rows landed logs a **Warning** naming the tag, the row count and the time range, so the orphan
can be found.
change without its trail row. A failed sample write after its trail rows landed logs a **Warning** naming the tag, the row count and the
time range. **Applied marker:** the trail row is written `pending = true` and, once the change has landed, re-upserted `pending = false` on
the table's `DEDUP UPSERT KEYS(ts, tag, modified_id)`. A modified read serves a still-`pending` row only when it is the **latest** trail row at
its time and the sample there (a tombstone included) was written at or after the modification — a crash between the change and its marker.
Otherwise it is the orphan of a change that never landed and is **not served** (it used to be served as a change that happened). Residual:
a double fault — a change that landed without its marker, followed by a later change of the same sample — hides the earlier one.
- **Rolling upgrade (SqlServer).** The v2 schema is applied by the first node that upgrades. Until its peer is upgraded too, that
not-yet-upgraded node still runs the old build, whose HistoryUpdates write **no trail rows** — silent gaps in the audit trail. Upgrade both
nodes of a pair together (stop, upgrade, start both); modifications made by an old-build node in the window are never recorded.
@@ -1116,11 +1119,15 @@ modification time and the Part 11 `HistoryUpdateType`. A HistoryReadRaw with `Is
decided at materialise time from "an updater is wired", and whether the schema has the trail is only known after the backend's lazy
first-use bootstrap (I/O the node manager does not do while materialising), so a v1 node is admitted to HistoryUpdate and then refused. Apply `002_sample_modified.sql` and restart the node to enable them (the verdict is taken once per
process). `AutoCreateSchema=true` (the default) still upgrades v1 to v2 in place. See [Schema](#schema).
- **QuestDB retry duplicates (#674).** `{p}sample_modified` is append-only with **no DEDUP key** (a key would collapse genuinely distinct
modifications of one sample), and each trail row gets a fresh `modified_id`. So a HistoryUpdate a client **retries** after a timeout — when the
first attempt had in fact landed — records a second trail row for the same change (and, for Insert, the retry is `BadEntryExists`). The
sample itself is idempotent (dedup upsert); only the audit trail can show the duplicate. `SqlServer` has the same exposure only if the client
retries a committed request; there is no request id to dedupe on in either backend.
- **QuestDB retry duplicates (#674).** `{p}sample_modified` carries **`DEDUP UPSERT KEYS(ts, tag, modified_id)`** (each modification keeps its
own `modified_id`, so distinct modifications of one sample never collapse). The updater re-sends a trail INSERT once, **with the same ids**,
after a transient failure (a client timeout that may have applied): it converges instead of duplicating. A HistoryUpdate the **client**
retries is a new request with new ids, so it still records a second trail row when the first attempt had in fact landed (and, for Insert,
the retry is `BadEntryExists`). `SqlServer` has the same client-retry exposure; there is no request id to dedupe on in either backend.
- **QuestDB trail migration.** A trail table from before the marker gains `pending` (old rows read `false`, so they stay visible) and the
dedup key on first boot with `AutoCreateSchema=true` (`ALTER TABLE … ADD COLUMN pending BOOLEAN`, `ALTER TABLE … DEDUP ENABLE UPSERT KEYS(ts,
tag, modified_id)`). Unlike the index ALTERs this one is awaited (≤ 10 s), because an INSERT naming the new column would fail before it
applies; until then, and with `AutoCreateSchema=false` (a Warning naming the statements), the old format is used — no marker, no retry.
- **Cross-node ordering (QuestDB).** The per-tag lock is per process; the [Primary-only](#primary-only-674) HistoryUpdate gate (#674) is what keeps
two nodes of a pair from recording concurrent updates of one tag (and so a wrong prior value). Two **separate** pairs sharing one QuestDB can
still race; that is unsupported.
@@ -1942,7 +1949,7 @@ makes HistoryUpdate Replace work; the writer's own insert-if-absent semantics ar
| `{p}sample` (`ts, tag, kind, num_value, int_value, str_value, status, deleted, server_ts`; **`tag` is a `SYMBOL INDEX`**, #683) | `(ts, tag)` | `PARTITION BY DAY` (TTL when configured) |
| `{p}event` (one-for-one with `hist.Event`; **`source` is a `SYMBOL INDEX`**) | `(ts, event_id)` | `PARTITION BY DAY` (TTL when configured) |
| `{p}tag` (`ts, name, data_type, eng_unit, description, updated_utc`) | `(ts, name)` with `ts` fixed at `1970-01-01` — an upsert by name | `PARTITION BY YEAR`, never expires |
| `{p}sample_modified` (`ts` = the sample's source time, `tag` (**`SYMBOL INDEX`**)`, modified_id, modification_ts, update_type, principal, api_key_id, kind, num_value, int_value, str_value, status, server_ts`) (#668) | none — append-only, no DEDUP; `modified_id` (process-monotonic ≈ UTC ticks) orders a timestamp's trail | `PARTITION BY DAY` (TTL when configured). A NULL `status` marks "no prior value" (QuestDB `BYTE` has no NULL, so `kind` is 0) |
| `{p}sample_modified` (`ts` = the sample's source time, `tag` (**`SYMBOL INDEX`**)`, modified_id, modification_ts, update_type, principal, api_key_id, kind, num_value, int_value, str_value, status, server_ts, pending`) (#668) | `(ts, tag, modified_id)` — each modification is its own row; the key only lets the applied marker and a re-sent INSERT converge. `modified_id` (process-monotonic ≈ UTC ticks) orders a timestamp's trail | `PARTITION BY DAY` (TTL when configured). A NULL `status` marks "no prior value" (QuestDB `BYTE` has no NULL, so `kind` is 0) |
**The `tag` index (#683).** Every read filters on one tag; without a symbol index QuestDB answers `tag = x` by scanning the tag
column of every partition in the time range, which made the processed-read [data-range queries](#aggregates-processed-reads) of a stopped or late-starting
@@ -2045,7 +2052,8 @@ present **and not deleted**.
**Modification trail (#668).** Each operation first appends one `{p}sample_modified` row per sample it will change, carrying the live row's
current value (read by the same existence query), then writes the change. No multi-table transaction exists, so this order is the guarantee:
a crash between the two leaves an **orphan trail row**, never an untracked change — the documented deviation from `SqlServer`'s atomic trail.
See [Modified-value history](#modified-value-history-part-11).
The row is written `pending` and marked applied after the change, so a modified read does not serve the orphan of a change that never
landed. See [Modified-value history](#modified-value-history-part-11).
The per-item codes come from the shared `HistoryUpdatePlanner`, so they equal `SqlServer`'s (including the duplicate-timestamp-in-one-request rules
above). **Every read and aggregate filters `NOT deleted`.** A client Insert at a deleted timestamp re-populates it (the OPC UA intent); only the