# Historian — server-side OPC UA HistoryRead / HistoryUpdate (pluggable backends) Phase C wires server-side OPC UA **HistoryRead** for authored equipment tags flagged historized. The feature is driver-agnostic: any equipment tag (Galaxy, Modbus, OpcUaClient, or any other driver) can be marked historized; the server dispatches all history reads to the registered `IHistorianDataSource` — the **HistorianGateway** read client (`GatewayHistorianDataSource`, talking gRPC to the external `ZB.MOM.WW.HistorianGateway` via the `ZB.MOM.WW.HistorianGateway.Client` package). No EF migration is required; the historian flag rides in the existing schemaless `TagConfig` JSON blob alongside the Phase B `alarm` object. (The bespoke Wonderware TCP sidecar backend this replaced was retired — see [drivers/Historian.Wonderware.md](drivers/Historian.Wonderware.md).) > **The backend is pluggable (2026-10).** `ServerHistorian:Backend` selects `Gateway` (the default — everything > described below as "the gateway"), `SqlServer` (a plain SQL Server database the server reads and writes > itself, which also supports OPC UA **HistoryUpdate**) or `QuestDb` (a QuestDB time-series database over the Postgres > wire protocol; also supports HistoryUpdate, via soft delete). See [Backend selection](#backend-selection-serverhistorianbackend), > [SQL Server backend](#sql-server-backend), [QuestDB backend](#questdb-backend) and [HistoryUpdate](#historyupdate). Wherever this page says > "the historian" without qualification it means the selected backend; sections that only apply to one backend say so. Design references: [docs/plans/2026-06-14-galaxy-phase-c-historian-design.md](plans/2026-06-14-galaxy-phase-c-historian-design.md) (Phase C) and [docs/plans/2026-10-09-pluggable-historian-design.md](plans/2026-10-09-pluggable-historian-design.md) (pluggable backends). --- ## Historized TagConfig schema A tag is historized by setting the **Historize this tag** checkbox and optional **Historian tagname (override)** textbox in the Tag modal on the `/uns` equipment page Tags tab. These controls work for **all drivers** — typed editors (Modbus, S7, OpcUaClient, etc.) and the raw-JSON textarea (Galaxy) alike. The controls merge `isHistorized` / `historianTagname` into the existing `TagConfig` JSON blob via the `TagHistorizeConfig` helper, preserving all other keys byte-stable. Drivers that still use the raw-JSON editor (Galaxy) can also add the fields directly in the textarea. ### Fields | Field | Type | Required | Description | |---|---|---|---| | `isHistorized` | bool | yes | Marks the tag historized. Materialises the OPC UA node with `Historizing=true` and the `HistoryRead` AccessLevel bit. | | `historianTagname` | string | no | Explicit historian tagname to query. When absent or empty, defaults to the tag's driver `FullName` (the `TagConfig.FullName`). | ### Examples Default historian tagname (uses `FullName` = `TestMachine_002.TestFloat`): ```json {"FullName":"TestMachine_002.TestFloat","isHistorized":true} ``` Explicit historian tagname override: ```json {"FullName":"TestMachine_002.TestFloat","isHistorized":true,"historianTagname":"Plant.Line1.Flow"} ``` A tag that is both historized and a native alarm: ```json {"FullName":"TestMachine_002.HiAlarm","isHistorized":true,"alarm":{"alarmType":"OffNormalAlarm","severity":700}} ``` --- ## Backend selection (`ServerHistorian:Backend`) `ServerHistorian:Backend` (case-insensitive, trimmed; default `Gateway`) picks the provider (`IHistorianBackendProvider`) that builds all five historian seams — the HistoryRead data source, the continuous- historization value writer, the alarm-history writer, the tag provisioner and (optionally) the HistoryUpdate updater. `HistorianBackendRegistry` in the Host owns the name table. An unknown name fails host start with an `OptionsValidationException` that lists the allowed values (`Gateway`, `SqlServer`, `QuestDb`), regardless of whether anything consumes the historian. The existing enable gates (`ServerHistorian:Enabled`, `AlarmHistorian:Enabled`, `ContinuousHistorization:Enabled`) are unchanged — they simply get their objects from the selected provider. | | `Gateway` (default) | `SqlServer` | `QuestDb` | |---|---|---|---| | Storage | AVEVA Historian via the HistorianGateway sidecar | `{Schema}` tables in a SQL Server database you point it at | `{prefix}sample` / `event` / `tag` / `sample_modified` WAL tables in a QuestDB instance | | Value kinds recorded | `Double`, `Int64`, `Boolean` (numeric analog only) | all: `Double`, `Int64`, `Boolean`, `String`, `DateTime`, plus `Empty` (a Bad sample with no value) | same as `SqlServer` | | Status code stored | Sent as DA quality but **discarded** by the gateway (C-002); AVEVA stores every sample Good | Stored bit-for-bit (`StatusCode` int column) | Stored bit-for-bit (unsigned, in a `LONG`) | | Comms-loss / driver-Bad publishes | Dropped, counted in `DroppedUnsupportedKind` | Stored as an `Empty` outage-marker sample carrying the Bad code (see [below](#typed-values-real-status-and-outage-markers)) | Stored as an `Empty` outage-marker sample, as on `SqlServer` | | Processed aggregates | Twelve: the ten per-sample ones (Average, Count, Minimum, Maximum, Range, Start, End, Delta, Min/MaxActualTime) evaluated from raw reads with the SQL backends' evaluator — identical results (#681) — plus TimeAverage and Total from AVEVA's native modes | The seventeen in the [aggregate table](#aggregates-processed-reads) (incl. Part 13 time-weighted, #671) | The same seventeen, identical results | | HistoryUpdate | No (`BadHistoryOperationUnsupported`) | Yes (Insert / Replace / Update / DeleteRaw / DeleteAtTime) | Yes — soft delete (tombstones), see [below](#historyupdate-soft-delete) | | Modified-value history (`IsReadModified`, #668) | No (`BadHistoryOperationUnsupported`) | Yes — `SampleModified`, atomic with the change | Yes — `{prefix}sample_modified`, written before the change (orphan-on-crash) | | Alarm history | gateway `SendEvent` | `Event` table (insert-if-absent) | `{prefix}event` table (dedup upsert on `(ts, event_id)`) | | Provisioning (`EnsureTags`) | numeric types only; others skipped | `MERGE` upsert of every request, all types | Upsert into `{prefix}tag`, all types | | Connection | `Endpoint` + `ApiKey` + TLS keys (top-level `ServerHistorian:*`) | `ServerHistorian:SqlServer:*` | `ServerHistorian:QuestDb:*` | Recording stays **Primary-only** (#537) and **at-least-once** with the #538 per-tag quarantine on every backend; HistoryRead is served from any node (the database is shared). HistoryUpdate is **Primary-only** since #674 (`BadInvalidState` on any other node — see [HistoryUpdate](#primary-only-674)). The historian connection string is **not** a ConfigDb string, and a driver-only node may carry it without breaking the Phase 4 "no ConfigDb" posture. --- ## ServerHistorian configuration The `ServerHistorian` section in `appsettings.json` controls the historian read path. `Enabled` defaults to `false`; when disabled, the server registers `NullHistorianDataSource` and Raw / AtTime / Events HistoryRead calls on historized nodes return `GoodNoData` (empty, not an error); a Processed read is `BadAggregateNotSupported` and no history aggregate functions are advertised ([#682](#advertised-aggregates-aggregatefunctions-682)). ```json { "ServerHistorian": { "Enabled": false, "Backend": "Gateway", "Endpoint": "", "ApiKey": "", "UseTls": true, "AllowUntrustedServerCertificate": false, "CaCertificatePath": null, "CallTimeout": "00:00:30", "MaxTieClusterOverfetch": 65536, "HonorReturnBounds": true, "GatewayAtTimeFromRaw": true, "EventReadHorizon": "00:00:00", "HistoryReadBatchParallelism": 4, "MaxConcurrentHistoryReadBatches": 16, "HistoryReadDeadline": "00:01:00", "MaxTimeWeightedRawRows": 1000000, "SqlServer": { "ConnectionString": "", "Schema": "hist", "CommandTimeout": "00:00:30", "AutoCreateSchema": true }, "QuestDb": { "ConnectionString": "", "TablePrefix": "hist_", "RetentionDays": null, "CommandTimeout": "00:00:30", "AutoCreateSchema": true, "VisibilityTimeout": "00:00:05" } } } ``` | Key | Type | Default | Description | |---|---|---|---| | `Enabled` | bool | `false` | Enable the selected backend's read client. `false` → `NullHistorianDataSource` (empty reads) and no HistoryUpdate. | | `Backend` | string | `Gateway` | `Gateway`, `SqlServer` or `QuestDb` (case-insensitive). Unknown ⇒ host start fails. See [Backend selection](#backend-selection-serverhistorianbackend). | | `Endpoint` | string | `""` | **Gateway backend only.** Absolute gateway URI, e.g. `https://host:5222`. Scheme selects transport (`https://` = TLS, `http://` = h2c plaintext). Required when `Enabled`. | | `ApiKey` | string | `""` | **Gateway backend only.** The gateway peppered-HMAC key (`histgw__`) sent as `Authorization: Bearer`. Required when `Enabled`. **Supply via env `ServerHistorian__ApiKey`.** | | `UseTls` | bool | `true` | **Gateway backend only.** Connect over TLS; must match the `Endpoint` scheme. | | `AllowUntrustedServerCertificate` | bool | `false` | **Gateway backend only.** Accept a self-signed / untrusted server certificate (dev / on-prem only). | | `CaCertificatePath` | string\|null | `null` | **Gateway backend only.** PEM CA file pinning the gateway's TLS chain. Null/empty uses the OS trust store. | | `CallTimeout` | TimeSpan | `00:00:30` | **Gateway backend only.** Per-call deadline applied to each unary gateway read. A read for a tag the historian does not yet know about — e.g. a freshly-historized tag that has **not** been provisioned (see [Tag auto-provisioning](#tag-auto-provisioning-ensuretags)) — can run to this **full** deadline before the gateway errors, so an unprovisioned-tag HistoryRead can block for ~30 s. Lower this to fail faster, at the cost of truncating legitimately long reads. | | `MaxTieClusterOverfetch` | int | `65536` | (Node-manager paging; all backends — a `SqlServer` PK rules out tie clusters, so it is moot there.) Maximum samples the server will fetch in one shot to page through a tie cluster (multiple samples sharing one `SourceTimestamp`). A cluster larger than this ceiling fails `BadTooManyOperations` (#686; was `BadHistoryOperationUnsupported`). Also the per-read row budget of a Gateway reverse raw read (walked back from forward reads, #686): one timestamp holding more samples is refused `BadTooManyOperations`. Values below 1 count as 1. Raise to handle abnormally large tie clusters; the default covers all normal-data cases. | | `HonorReturnBounds` | bool | `true` | (#691, all backends.) Whether HistoryRead-Raw honours `ReturnBounds` (Part 11 bounding values, [Bounding values](#bounding-values-returnbounds-691)). `false` is the kill switch back to the pre-#691 behaviour: the flag is ignored (stored samples only) and a modified read carrying it is not refused `Bad_InvalidArgument`. Note the .NET SDK defaults both `ReturnBounds` and `IsReadModified` to `true`. | | `GatewayAtTimeFromRaw` | bool | `true` | (#693, **Gateway backend only.**) Whether HistoryRead-AtTime is served from raw gateway reads evaluated with the SQL backends' AtTime rules ([Gateway AtTime](#gateway-attime-is-evaluated-from-raw-reads-693)). `false` is the kill switch back to the pre-#693 answer: AVEVA's own interpolation (the gateway's `ReadAtTime`) passed through unchanged. | | `EventReadHorizon` | TimeSpan | `00:00:00` | (#694, all backends, **opt-in, lossy**.) `00:00:00` = no horizon: every HistoryReadEvents read is exact to its own window. A positive span raises the earlier bound of every fresh events window to `now − EventReadHorizon` (the floor instant itself included), in **both** directions — a window wholly older is `GoodNoData` and is not sent; a page-through keeps the first page's floor. It exists for the Gateway, where a walked reverse / end-only read proves "nothing older" down to the Unix epoch (~3.8 s live; an unfiltered one can hit the gateway's 30 s SQL timeout) and a forward read from a far-past `StartTime` scans the same range. ⚠️ While set, events older than the horizon are **invisible** to HistoryReadEvents. Raw/Processed/AtTime are unaffected. Negative ⇒ treated as none (a `Validate()` warning). See [Continuation-point paging (Events)](#continuation-point-paging-events-669). | | `HistoryReadBatchParallelism` | int | `4` | Max historized nodes served **concurrently** within one HistoryRead batch (arch-review 03/S3). The SDK's `HistoryRead*` override surface is synchronous, so each arm bridges once at its boundary; this turns per-node serving from sequential (`N × CallTimeout` worst case) into `⌈N / P⌉ × CallTimeout`. Also caps the gateway-load multiplication a single client can command. `≤ 1` serves sequentially. | | `MaxConcurrentHistoryReadBatches` | int | `16` | Process-wide limit on HistoryRead (and HistoryUpdate) batches served **concurrently** (arch-review 03/S3). A flood of history clients degrades to `BadTooManyOperations` rather than exhausting the SDK request-thread pool. With `HistoryReadBatchParallelism` this caps total in-flight gateway reads at `MaxConcurrentHistoryReadBatches × HistoryReadBatchParallelism` (default 16 × 4 = 64). A batch that cannot get a slot within `min(HistoryReadDeadline, 5 s)` fails every handle `BadTooManyOperations`. | | `MaxTimeWeightedRawRows` | int | `1000000` | `SqlServer` / `QuestDb` (#671), and the `Gateway` per-sample aggregates (#681 — every per-sample read there streams raw samples, so the cap applies to the window's raw rows; see [Aggregate set](#aggregate-set-and-badaggregatenotsupported-285)). The most raw in-window samples one [time-weighted aggregate](#time-weighted-aggregates-part-13-671) read (TimeAverage, TimeAverage2, **Total**, Total2, Interpolative, and — since #680 — **PercentGood/PercentBad**) may stream into the server-side evaluator — that family is **not** pushed down to SQL. The window query is capped at this + 1 rows; one row past the cap fails **that node** with `BadTooManyOperations` (not counted as a backend fault) instead of holding a pooled connection until `HistoryReadDeadline` and returning `BadTimeout`. Bound rows outside the window do not count; per-sample aggregates are not capped on the SQL backends (pushed down). 1 000 000 ≈ 11.5 days of a 1 Hz tag. Non-positive = unbounded (a `Validate()` warning). `Gateway` (#688): the row budget it asks the gateway for (`cap + 18`) is clamped at the gateway's default `MaxValuesCeiling` (1,000,000) while the cap is at or under it, so the default works against a gateway with `AllowStreamingBeyondCeiling=false`; against a gateway with a *lower* ceiling the refused budget is retried once at the ceiling the gateway names (#691.3), so no change is needed (`ceiling − 18` only saves that round-trip). | | `HistoryReadDeadline` | TimeSpan | `00:01:00` | Per-request wall-clock deadline for a HistoryRead (and HistoryUpdate) batch (arch-review 03/S3). A node still in flight at the deadline surfaces `BadTimeout`, so one request can never hold an SDK request thread longer than this regardless of node count. Non-positive = unbounded (a `Validate()` warning). | | `HistoryUpdateMaxFutureSkew` | TimeSpan | `1.00:00:00` | #674. How far past the server clock a HistoryUpdate (Insert / Replace / Update) item's `SourceTimestamp` may lie; an item beyond `now + skew` is refused **`BadOutOfRange`** by the node manager before any backend sees it (every backend). Guards against a far-future typo (a 2062 timestamp). The live recorder is not bounded. Non-positive disables the bound (no warning — a deliberate choice). See [HistoryUpdate](#historyupdate). | **`ServerHistorian:SqlServer`** (read only when `Backend=SqlServer`; bound by `HistorianBackendRegistry.BindSqlServer`): | Key | Type | Default | Validation | Description | |---|---|---|---|---| | `SqlServer:ConnectionString` | string | `""` | **Required** (non-blank) | ADO.NET connection string for the historian database. Never the ConfigDb. Supply via `${secret:OT-HISTORIAN-SQL}` / environment (`ServerHistorian__SqlServer__ConnectionString`) — never commit a real one. The database must already exist; `AutoCreateSchema` creates the *schema and tables*, not the database. | | `SqlServer:Schema` | string | `hist` | Must match `^[A-Za-z_][A-Za-z0-9_]{0,127}$` (it is spliced into DDL/DML) | SQL schema that holds `SchemaVersion`, `Tag`, `Sample`, `Event` and the `SampleRow`/`TimeRow` table types. | | `SqlServer:CommandTimeout` | TimeSpan | `00:00:30` | Must be `> 0` | Per-command timeout, rounded up to whole seconds (minimum 1 s). The SQL backend does **not** use `CallTimeout`. | | `SqlServer:AutoCreateSchema` | bool | `true` | — | `true`: on first use apply any missing embedded script under a `sp_getapplock` (`OtOpcUaHistorianSchema`), so concurrent nodes serialise. `false`: only verify `SchemaVersion >= 1`; if behind, **every** historian call fails with a clear error naming the missing version (surfaced in the read client's health `LastError`, never a host crash). A failed bootstrap is retried on the next call, not latched. | | `SqlServer:RetentionDays` | int? | unset | `>= 1` when set | Days of history kept, on **source time**. Unset keeps everything. When set, driver nodes purge older `hist.Sample` / `hist.Event` rows — see [Retention](#retention-retentiondays). | | `SqlServer:PurgeInterval` | TimeSpan | `01:00:00` | `> 0` | How often the purge runs (first pass at start-up). | | `SqlServer:PurgeBatchSize` | int | `4000` | `>= 1` | Rows per `DELETE TOP (n)`; the default stays under SQL Server's ~5000-lock escalation threshold so a purge never takes a table lock. | **`ServerHistorian:QuestDb`** (read only when `Backend=QuestDb`; bound by `HistorianBackendRegistry.BindQuestDb`): six keys — `ConnectionString` (required), `TablePrefix`, `RetentionDays`, `CommandTimeout`, `AutoCreateSchema`, `VisibilityTimeout`. The full table, defaults and validation rules are in [QuestDB backend → Configuration](#configuration-serverhistorianquestdb) (and [Configuration.md](Configuration.md#serverhistorianquestdb-backendquestdb)). **Validation rules** (`ServerHistorianOptionsValidator`, driver-role nodes only; admin-only nodes never build historian singletons): - `Backend` must name a registered backend — checked **always**, with no I/O, even when nothing is enabled. - Everything else is **consumer-gated**: it runs only when `ServerHistorian:Enabled` **or** `AlarmHistorian:Enabled` is true (the alarm sink sources its connection from this section even when `ServerHistorian` is off). - `Backend=Gateway`: `Endpoint` must be an absolute http(s) URI, `ApiKey` non-empty, and `UseTls` must agree with the endpoint scheme. - `Backend=SqlServer`: the three `SqlServer` rules above. `Endpoint`/`ApiKey`/`UseTls` are **not** required. - `Backend=QuestDb`: `QuestDbHistorianOptions.Validate()` (connection string required, prefix regex, `RetentionDays >= 1`, `CommandTimeout > 0`, `VisibilityTimeout >= 0`). `Endpoint`/`ApiKey`/`UseTls` are **not** required. A malformed `TablePrefix` is also rejected by the connection helper when the provider first builds a seam. - ⚠️ `Backend=SqlServer` with a malformed `Schema` throws `ArgumentException` while the provider is built, so it stops host start even when no consumer is enabled (the validator is consumer-gated, provider construction is not). - The alarm-only warning at startup (`AlarmHistorian` on, `ServerHistorian` off) names the selected backend. > **Do not commit `ApiKey` to `appsettings.json`.** Set it via the environment variable > `ServerHistorian__ApiKey`, a secrets store, or a deployment-time overlay. The checked-in default is > always empty. > **Gateway-side prerequisites.** The target gateway must run `RuntimeDb:Enabled=true` (continuous > `WriteLiveValues`) + `RuntimeDb:EventReadsEnabled=true` (alarm-history `ReadEvents`), and the API key > must carry the scopes `historian:read`, `historian:write`, `historian:tags:write`. > **Migration from the Wonderware backend.** Rename the old keys: `Host`/`Port` → `Endpoint` > (`https://host:5222`); `SharedSecret` → `ApiKey` (env `ServerHistorian__ApiKey`); > `ServerCertThumbprint` → `CaCertificatePath` (+ `UseTls` / `AllowUntrustedServerCertificate`). The `ServerHistorian` section is independent of the `AlarmHistorian` section (the alarm write path) and the `ContinuousHistorization` section (driver-value capture). All three target the **same** gateway — but only `ServerHistorian` carries the connection (endpoint/key/TLS); the other two source it from there. > **Continuous-historization value quality on the Gateway backend is still server-stamped.** The recorder now > carries each sample's **real** OPC UA status (it used to hard-code Good `192`, so a Bad sample was historized as > Good). `GatewayHistorianValueWriter` maps that status to an OPC-DA quality byte with > `GatewayQualityMapper.ToDaQuality` (the inverse of the read-side table; unknown codes fall back by severity: > Good ⇒ 192, Uncertain ⇒ 64, Bad ⇒ 0). But the gateway's `WriteLiveValues` SQL path **discards** the quality > it is sent (a server-managed column; 0.2.0 Contracts C-002), so **AVEVA still stores every recorded value as > Good** — a pre-existing limitation of that backend, now documented rather than fixed (archreview 06/C-7). The > `SqlServer` backend stores the real status. > ✅ **Continuous value capture is LIVE-PROVEN END-TO-END > ([#491](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/491) CLOSED — gate run 2026-08-01, > completed 2026-08-04; see `docs/plans/2026-08-01-continuous-historization-live-gate.md`).** Against a > real `ZB.MOM.WW.HistorianGateway` talking to the real AVEVA Historian on `wonder-sql-vd03`: deploy → > `EnsureTags` provisioning → dependency-mux value delta → FasterLog outbox append → drain → > `WriteLiveValues` → **value read back from the historian**, plus **restart convergence** (the recorder > re-registers the deployed ref set after a process restart, with no new deployment). > > The read-back used two registers carrying **distinct non-zero** constants (`8224`, `120`), each of > which read back as its own value — a `0`-valued source cannot distinguish a correct value from a > wrong one. > > ⚠️ The gate found a **production-fatal defect**: the recorder threw `NotSupportedException` on the > **first value it ever appended**, because it touched `Self` after an `await …ConfigureAwait(false)`, > which discards Akka's ActorContext. Supervision restarted the actor, and since the recorder is seeded > with an *empty* ref set, the restart discarded every historized ref — historization stopped after one > value. Fixed; pinned by a regression test whose outbox fake completes on a **pool** thread (a fake > returning `ValueTask.CompletedTask`, or even `await Task.Yield()`, cannot reproduce it). > > The gate also found the two defects below, both since fixed. > > **Related, separately open:** `EnsureTags` provisioning fires only for **newly-added** historized > tags (`AddressSpaceApplier` iterates the added set), so flipping an **existing** tag to > `"isHistorized": true` never provisions it — see > `docs/plans/2026-06-27-historian-gateway-integration-issues.md` Issue 5. --- ## ContinuousHistorization configuration `ContinuousHistorizationOptions` binds the `ContinuousHistorization` section. It carries **only** the recorder + outbox knobs — the backend connection (gateway endpoint / key / TLS, or the SQL Server connection string) is sourced from `ServerHistorian` (above), never restated here. The outbox is a node-local FasterLog and is registered the same way for every backend. | Key | Type | Default | Description | |---|---|---|---| | `Enabled` | bool | `false` | `true` **and** a configured `ServerHistorian` wires the FasterLog outbox + the selected backend's value writer and spawns the `ContinuousHistorizationRecorder`. `false` → no recorder; driver tag values are not historized. | | `OutboxPath` | string | `""` | **Directory** holding the FasterLog segment + commit files. Required when `Enabled`. In production set an **absolute** path on durable storage — a relative path resolves against the host's working directory. | | `CommitMode` | string | `PerEntry` | `PerEntry` fsyncs before the append returns (no loss window); `Periodic` batches commits onto a background timer every `CommitIntervalMs`. Parsed case-insensitively; an unrecognized value falls back to `PerEntry`. **See the batch-commit note below — `PerEntry` is one fsync per ingest *burst*, not per value.** | | `CommitIntervalMs` | int | `100` | Periodic-mode commit cadence (ms); must be `> 0` under `Periodic`, ignored under `PerEntry`. | | `DrainBatchSize` | int | `1024` | Outbox entries peeked + written per drain pass. **Raised from 64**: the peek is an in-memory scan of the outbox's live index (widening it costs allocation, not I/O), and a pass is now bounded-parallel over `DrainConcurrency` rather than sequential, so a wide batch no longer costs `tags × latency`. At 64 a drain drew down at most 64 entries every `DrainIntervalSeconds` — below the arrival rate of even a modest historized tag set — so any gateway hiccup left the outbox permanently behind, growing toward `Capacity` and shedding oldest samples instead of catching up. | | `DrainIntervalSeconds` | double | `2` | Steady drain cadence (and the post-success reschedule). | | `DrainConcurrency` | int | `8` | Historian tags a single drain pass writes **concurrently**. One peeked batch is grouped per historian tag and each group is an independent gateway round-trip, so a sequential pass cost `tags × latency`; this window makes it `⌈tags / concurrency⌉ × latency` while bounding how many round-trips can be outstanding against the gateway at once. `1` restores the fully sequential drain; a value `< 1` is clamped to 1 (with a `Validate()` warning). **The ack is unchanged** — still all-or-nothing after every group has reported, and the #538 per-tag quarantine still keys on "another group wrote this pass". | | `Capacity` | int | `500000` | Max un-acked outbox entries before the drop-oldest policy sheds the front of the queue; `0` = unbounded. Roughly 50 MB of FasterLog. The #538 **disk backstop** for the one case the per-tag quarantine deliberately cannot resolve — a historian unreachable or refusing *every* write, where nothing proves any particular tag is at fault so the recorder correctly retains. | | `MaxWriteAttempts` | int | `5` | #538 poison-tag budget: consecutive failed drain passes one historian tag may accumulate before its queued entries are abandoned and the FIFO head is allowed to advance. **Counted only across passes where *other* tags wrote successfully**, so an ordinary gateway outage never advances it. Clamped to at least 1; each abandonment logs a Warning naming the tag. | | `CaptureWhilePrimaryUnreachable` | bool | `true` | #672: a standby whose capture is stood down for its mesh-partner Primary (#537) resumes capturing as soon as the cluster failure detector reports that Primary **unreachable**, instead of waiting for auto-down + a fresh redundancy snapshot. Shrinks a crash-failover capture gap from ~25 s to the failure-detector window (~10–12 s), at the cost of short duplicate windows — see [the crash-failover note](#a-crashed-primary-capture-gap-672). `false` restores the pre-#672 behaviour. Never affects the alarm drain or the Primary data-plane gate. | | `MinBackoffSeconds` | double | `1` | Initial retry backoff after a failed drain pass. | | `MaxBackoffSeconds` | double | `30` | Cap on the exponential retry backoff. | > **`PerEntry` now means one fsync per ingest *burst*, not per value.** The recorder appends a whole > value burst through `IHistorizationOutbox.AppendBatchAsync`, which enqueues every entry and then > issues a **single** group commit — so a burst of N values costs one fsync rather than N. The > durability contract is unchanged: the call does not return until that commit completes, so nothing > the caller has been told is buffered can be lost to an unclean restart. (The documented residual is > unchanged too: a throw *before* the commit leaves the enqueued prefix tracked-but-uncommitted — > served correctly for the life of the process, absent after an unclean restart — exactly the > enqueue-then-commit window the single-entry path always had, widened from one entry to the batch.) > **Cross-repo follow-up — a multi-tag batch `WriteLiveValues` on the gateway contract.** The drain's > unit of work is one gateway round-trip **per historian tag**, because > `ZB.MOM.WW.HistorianGateway.Client` **0.3.0** exposes no RPC that writes values for more than one tag > in a call. Bounded parallelism (`DrainConcurrency`) is therefore the ceiling: it overlaps the > round-trips but cannot remove them, so a wide tag set still pays `⌈tags / concurrency⌉` network > hops per drain pass. Collapsing that to a single hop needs a **batch/multi-tag `WriteLiveValues`** on > the `historian_gateway.v1` contract in the `~/Desktop/HistorianGateway` sister repo. Until that > lands, raising `DrainBatchSize` widens each hop's payload but not the hop count — which is why the > concurrency window, not the batch width, is what governs drain throughput today. ### Typed values, real status and outage markers The recorder no longer assumes "a numeric Good sample". Each mux delta is coerced to a typed `HistorianValue` (`Double` / `Int64` / `Boolean` / `String` / `DateTime` / `Empty`; a `ulong` above `long.MaxValue` becomes `Double`; arrays and anything else are dropped and counted) and appended with the **real** status the driver published. A value whose kind is not in the selected backend's `SupportedKinds` is dropped at capture and counted in `RecorderStatus.DroppedUnsupportedKind` (the field was `DroppedNonNumeric` before the rename; `Empty` is supported by `SqlServer`, not by `Gateway`). ⚠️ **Comms loss is historized — as an outage marker (design §9.10).** `DriverHostActor.FanCommsLossToDependencyMux` (#481) fans `(null, BadNoCommunication)` for every value ref of a comms-lost driver to the dependency mux, and the recorder cannot tell that from a driver's own Bad publish. So: - **`SqlServer` and `QuestDb`:** each disconnect stores **one `Empty` / `BadNoCommunication` (`0x80310000`) sample per historized tag**. A driver's own Bad publish is stored the same way (`Empty` kind plus its own code — e.g. Modbus `BadCommunicationError` `0x80050000`). This is deliberate: raw reads show the gap, and aggregates and AtTime interpolation do not bridge it. Cost: one extra row per tag per outage. - **`Gateway`:** the marker is `Empty`, which that backend does not support, so it is dropped and counted in `DroppedUnsupportedKind`. ### Outbox format v2 and rollback Outbox entries are written in format **v2**: `"HV2\0"` magic + version byte `2` + id + tag + value kind + a kind-specific payload + the 32-bit status + timestamp. Entries left in an outbox by an older build (the fixed 38-bytes-plus-tag layout, always quality 192) are still **read** — as a `Double` with status `Good`. A record that matches neither layout is **skipped and counted, never thrown**, so a corrupt record cannot jam the drain (this also holds during the FasterLog recovery scan at start-up); the lifetime count is the gauge **`otopcua.historian.outbox.unreadable`** (`IHistorizationOutbox.UnreadableCount`). A growing value means undecodable records are being skipped. > **Metric prefixes differ.** The outbox's other two gauges are `otopcua.historization.outbox.depth` and > `otopcua.historization.outbox.dropped`; the unreadable gauge is **`otopcua.historian.outbox.unreadable`** > (the name the design fixed). All three describe the same outbox — search for both prefixes. ⚠️ **Rollback: drain or delete the outbox before downgrading.** A node running a pre-v2 build cannot read v2 records — its outbox deserializes every record in its constructor and fails to start on the first v2 one. Before rolling a node back, let the outbox drain to zero (`otopcua.historization.outbox.depth`) or stop the node and delete the directory named by `ContinuousHistorization:OutboxPath` (accepting the loss of anything un-drained). Upgrading needs no action. **Delivery semantics on `SqlServer`** (`QuestDb` is the same, at µs resolution and with tombstones counting as present — see [QuestDB backend](#live-writes-insert-if-absent-tombstones-included)). Writes are *insert-if-absent* on `(TagId, SourceTimeUtc)`: the outbox's at-least-once redelivery (and both pair nodes writing the same sample in a boot window) converges on one row, and the first-stored sample wins. **Duplicate timestamps inside one batch collapse to the first.** A `NaN` / `±Infinity` `Double` cannot be stored in a SQL `float`, so the **value writer stores it as an `Empty` sample** — the tag is not quarantined for one bad reading (HistoryUpdate behaves differently: see [below](#historyupdate)). ⚠️ **An `Empty` sample is always Bad** (the quality contract: a stored sample is never served better than "no value"): a non-finite `Double` that arrived with a Good or Uncertain status is stored as `Empty` + **`BadOutOfRange`** (`0x803C0000`), and any other `Empty` with a non-Bad status as `Empty` + **`BadNoData`** (`0x809B0000`). An existing Bad code — the comms-loss `BadNoCommunication` marker, a driver's own Bad publish — is stored as it is. A duplicate-key error (2627/2601) rolls the whole statement back, so the writer re-runs the statement (at most twice) instead of treating the error as success. --- ## A poison tag no longer pins the outbox (#538) `DrainOnceAsync` peeks a batch, writes **every** per-tag group in it, and then acks **all-or-nothing** — because the outbox's `RemoveAsync` truncates the FIFO head *through* the acked id, so acking one tag's entries could silently drop older, unwritten entries belonging to another. That much is deliberate and unchanged. What was wrong is what happened when a tag failed **permanently**: the head never advanced, so every **healthy** tag in the batch was re-submitted to the historian on every drain pass, indefinitely — duplicate samples at identical timestamps, and an outbox that never truncated (observed live at 8.3 MB from a single unprovisionable tag, with the gateway's `ok` **and** `error` counters climbing together — the signature of this bug). The drain now tracks **consecutive failures per historian tag** and abandons a tag's queued entries once it exceeds `ContinuousHistorization:MaxWriteAttempts` (default 5), letting the head advance. Two properties matter: - **A pass where NOTHING wrote abandons nothing.** The counter only advances on a pass where at least one *other* tag wrote successfully — that is what distinguishes "this tag is poison" from "the backend is down". A count-only quarantine looks identical on the poison-tag case and silently discards the whole backlog during an ordinary gateway restart, turning a duplicate-samples bug into a lost-history one. Pinned by a falsification-control test (`A_backend_that_fails_every_tag_abandons_nothing`) that goes red when the evidence check is removed while the poison-tag test stays green. - **A tag over budget waits for the others.** Abandonment is deferred until *every* failing tag in the batch is over budget, so a tag is never discarded on a pass that could not have advanced the head anyway. Each abandonment logs a Warning naming the tag and the discarded sample count, and the total rides `RecorderStatus.AbandonedEntries`. The most common cause is a tag the historian has no definition for — easy to reach, because `EnsureTags` provisioning runs only for newly-**added** historized tags, so a changed `historianTagname` is never provisioned (Issue 5 above). `ContinuousHistorization:Capacity` now also defaults to **500,000** entries rather than unbounded. It is the disk backstop for the one case the quarantine deliberately cannot resolve — a historian refusing *every* write, where retaining is correct. Set `0` to restore unbounded retention. ## Only the recording Primary captures (#537) Both nodes of a redundant pair serve the same address space and see the same dependency-mux fan-out, so an ungated recorder wrote **every sample twice** (measured: the gateway's write counter advanced twice per value with both centrals up, and halved when one was stopped). The recorder now reads a capture gate per value, fed from `PrimaryGatePolicy.ShouldRecordHistorizedValues` via `IHistorizationRoleView` — the same shape as the alarm store-and-forward sink's `drainGate`, with two deliberate differences: - **It gates the CAPTURE, not the drain.** The alarm queue lives in LocalDb and *replicates*, so the Secondary already holds the Primary's rows and gating the drain loses nothing. The historization outbox is a node-local FasterLog replicating to nobody: gating only its drain would leave the standby appending forever to a queue no one empties — unbounded disk, and a flood of stale duplicates the moment it is promoted. Entries a node already captured are still drained by that node, so a demotion loses nothing recorded. - **The relevant peer is one that records the same refs**, not one that holds the same rows. A node stands down only for a Primary in its own per-cluster mesh. A node carrying no cluster-scoped role cannot establish that, so it never stands down: under a fleet-wide election the elected Primary is usually running a different cluster's configuration, and deferring to it would end this cluster's historization silently — the drain would report success on an empty outbox forever. That is the same scope mismatch that once suspended the alarm drain on every node of the docker-dev rig at once. An unknown role **captures**, matching the alarm drain: a duplicate sample is recoverable, a hole in the trend is not. The residual cost is a capture gap across a failover, bounded by how long the newly promoted node takes to see its first redundancy snapshot — inherent to any non-duplicating design. Suppressed values are counted in `RecorderStatus.SuppressedNotPrimary`, and each gate transition logs once. ### A crashed Primary: capture gap (#672) On a **crash** (SIGKILL / `docker kill`) that "first snapshot" can be a long way off. The dead Primary stays a cluster member — *unreachable, not downed* — for the failure-detector window plus `auto-down-unreachable-after` (`acceptable-heartbeat-pause` 10 s + 15 s in `akka.conf`), and the redundancy singleton almost always lived on it (it is the oldest member), so no snapshot is published until auto-down re-homes the singleton. The standby's cached snapshot keeps naming the dead node Primary and **nobody captures** for ~30 s (measured on the QuestDB live gate). `DriverHostActor` therefore follows cluster reachability (its own `ClusterEvent.ReachabilityEvent` subscription) and, with `ContinuousHistorization:CaptureWhilePrimaryUnreachable` on (the default), **excludes an unreachable peer from the historization "a mesh-partner Primary is recording" question**, re-deriving and republishing only the capture verdict from the last snapshot. A node at the same address coming Up again clears the mark; a reachable-again Primary closes the gate again. It logs `capturing historized values while recording Primary is unreachable` once on entry and once on revert. The alarm drain and the Primary data-plane verdict (writes, native acks, the Sparkplug host claim) are **not** touched — a false allow there is a dual claimant, not a duplicate sample. Trade-offs, all in the duplicate direction #537 already accepts: - **The gap shrinks to the failure-detector window** (~10–12 s), not to zero — detection is the floor. - **Duplicates during the overlap.** A Primary that is only *briefly* unreachable (GC pause, network blip) overlaps the standby until it is reachable again, and under a genuine **partition** both sides capture from detection onward (as they already did after auto-down, now ~15 s earlier). SqlServer and QuestDB absorb a duplicate of a **source-timestamped** sample idempotently (`(TagId, SourceTimeUtc)` key / dedup), but a driver with **no source timestamp** (e.g. Modbus: `SourceTimestampUtc` is null, so each node stamps its own poll time) writes a duplicate row on **every** backend, and the **Gateway** backend stores duplicates of everything. - Set `CaptureWhilePrimaryUnreachable=false` to keep the pre-#672 behaviour (no duplicates from this path, the full ~30 s hole on a crash). --- ## Tag auto-provisioning (EnsureTags) When a deploy adds a historized tag, the server auto-ensures it exists in the historian via the gateway's `EnsureTags` (Gateway) or the `Tag`-row `MERGE` (SqlServer) **before** the first value or HistoryRead lands. `AddressSpaceApplier.Apply()` fires a **non-blocking, fire-and-forget** `IHistorianProvisioning.EnsureTagsAsync` for each added historized value tag (with the resolved historian name — the `historianTagname` override when set, else the driver `FullName`); the hook is wrapped so a faulted, slow, or throwing provisioner can **never** block or fail a deploy. On the **Gateway backend**, non-numeric (`String`/`DateTime`/`Reference`) types are skipped, not provisioned; the SqlServer backend provisions every type (below). **Wiring + gating.** The concrete provisioner comes from the selected backend's provider (`CreateProvisioning`): the gateway-backed `GatewayTagProvisioner`, or — for `SqlServer` — `SqlServerTagProvisioner` (one `MERGE … WITH (HOLDLOCK)` upsert per chunk of 400 tags; every data type is provisionable so `skipped` is always 0; a blank or over-400-character name fails alone; a null unit/description in a request keeps the stored value). It is registered by the Host's `AddHistorianProvisioning` and gated on the **same** `ServerHistorian:Enabled` flag as the read path — both use the one resolved backend. When the section is disabled (or absent) the applier resolves the no-op `NullHistorianProvisioning` and **provisioning does not run** (so on a read-only-historian deployment, ensure tags exist by some other means). On the Gateway backend the API key must carry the `historian:tags:write` scope for `EnsureTags` to succeed. **Observability.** Every dispatch logs an Information tally — `historian provisioning completed (dispatched=N, requested=…, ensured=…, skipped=…, failed=…)`. A line showing `dispatched=N` but `requested=0` indicates the **no-op** provisioner ran (provisioning is not wired for this node), which is the signal to check that `ServerHistorian:Enabled=true`. > **Why this matters for HistoryRead latency.** Until a tag is provisioned, the historian does not know it, > so a HistoryRead on it runs to the full `CallTimeout` (~30 s) before erroring. Auto-provisioning on deploy > keeps freshly-historized tags from hitting that slow path. See the `CallTimeout` row above. ### Deferred types and the UInt16 fallback (#600 item 4) > **Gateway backend only.** The `SqlServer` backend provisions and records every type (`String` and `DateTime` included). Historization on the **Gateway** backend is **numeric-analog only**. `HistorianTypeMapper` classifies exactly nine types as historizable (`Boolean`, `Int16/32/64`, `UInt16/32/64`, `Float32`, `Float64`): - **Deferred: `String`, `DateTime`, `Reference`.** The historian's SQL analog write path cannot carry them. Provisioning skips them (counted in the `skipped=` tally, via `IsHistorizable`), the mapper throws `NotSupportedException` for them, and `ContinuousHistorizationRecorder` drops their values and counts them in `DroppedUnsupportedKind` (renamed from `DroppedNonNumeric`). A tag authored `isHistorized` with one of these types therefore silently records nothing; HistoryRead on it returns whatever the historian holds (typically `GoodNoData`). - **Fallback: `UInt16 -> Uint4`.** The historian's `UInt2` write path is unproven upstream, so UInt16 tags are provisioned as the wider `UInt4`. Values are unaffected; only the historian-side storage type differs. (`Boolean -> Int2` is the other fallback, below.) Lifting either needs gateway-side support first; neither is scheduled. ### Boolean historization maps to Int2 (issue #441) `EnsureTags` is contract-documented (HistorianGateway 0.2.0+) as an **upsert** — *"creates or updates historian tags according to the given definitions"* — that maps per-tag onto AVEVA's `HistorianClient.EnsureTagAsync(definition) -> bool`. It is **not idempotent** (not routed through the idempotent retry pipeline) and **not skip-if-exists**. `HistorianTypeMapper` maps **`DriverDataType.Boolean → HistorianDataType.Int2`** and Boolean values historize as a small **0/1 integer**. Int2 is used rather than the semantically-exact `Int1` because the historian's **`Int1` analog-tag creation path is server-degenerate** — `EnsureTags(Int1)` stores an unusable stub, so a Boolean tag mapped to `Int1` never becomes historizable. This is an evidence-based decision from a live probe against the deployed 2023 R2 historian; the root cause and gateway-side resolution are tracked in [issue #441](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/441) and HistorianGateway [#11](https://gitea.dohertylan.com/dohertj2/historiangw/issues/11) / PR #16 (which documents `Int1` as an explicit, evidence-based exclusion). The value-write path is unaffected: the recorder sends every value as a `double` on `WriteLiveValues` regardless of tag type, so a Boolean `0.0/1.0` round-trips cleanly into the Int2 tag. **Observed live (2026-07-13, real 0.2.0 gateway → real AVEVA historian on `wonder-sql-vd03`, probed via `grpcurl` — archreview R2-06 T12):** - `EnsureTags` is a **real create-or-update upsert** — re-`EnsureTags` of an existing tag at the same type returns `Success=true`. - An **in-place retype between SUPPORTED types is ACCEPTED** — `Float→Double` on an existing tag returns `Success=true`. - A type sweep confirmed **`Int2/UInt2/Int4/UInt4/Float/Double` all provision `Success=true`; only `Int1` fails** (`Success=false, Error="ProtocolEvidenceMissingException"` — the vendored client's "no capture evidence for this type" guard → gRPC `Unimplemented`), both for a fresh tag and as a retype target. Mapping Boolean to `Int2` sidesteps that gap entirely. > **Prerequisite for provisioning to land at all:** the provisioner must stamp a **positive `StorageRateMs`** > — a `0` made `EnsureTags` throw `ArgumentOutOfRangeException` and fail *every* provision (fixed in > `GatewayTagProvisioner`, PR #439). **Migration note for pre-existing Boolean tags.** A Boolean tag historian-side as Float (e.g. from the 0.1.0-era default) is asked, on its next deploy, to upsert to Int2. Both are supported analog types, so the retype is accepted. Provisioning is fire-and-forget and never blocks/fails a deploy; a per-tag rejection (if any) surfaces only as the tally's `failed` count, and values written to a still-Float tag remain readable (Float supersets the 0/1 range) pending reconcile. A bulk retype sweep is **not** done in code. --- ## HistoryRead behavior ### Read variants The server supports all four OPC UA HistoryRead variants: | Variant | Node type | CLI `--aggregate` | |---|---|---| | **Raw** | Historized variable | (omit `--aggregate`) | | **Processed** | Historized variable | `Average`, `Minimum`, `Maximum`, `Count`, `Start`, `End`, `StandardDeviation` (the CLI's own list; `SqlServer` / `QuestDb` serve seventeen aggregates — see the [aggregate table](#aggregates-processed-reads)) | | **AtTime** | Historized variable | n/a (client supplies exact timestamps) | | **Events** | Equipment folder (event notifier) | n/a | | **Modified** (Raw with `IsReadModified = true`, #668) | Historized variable | `--modified` (see [Modified-value history](#modified-value-history-part-11)) | **Variable nodes** (historized tags) serve Raw, Processed, and AtTime history. `Historizing=true` and `AccessLevels.HistoryRead` are set at materialization so any compliant OPC UA client can discover historized capability from the node's attributes. > **v3 Batch 4 — dual-namespace registration.** A historized raw tag surfaces as **two** > variable nodes: the Raw-namespace node (`ns=Raw, s=`) and, for each referencing > equipment, a UNS-namespace node (`ns=UNS, s=///`). > **Both NodeIds register the SAME historian tagname** (the raw tag's `FullName` / > `historianTagname`), so HistoryRead against either NodeId resolves to the same tagname and > returns the same series. The historization intent is single-sourced — the mux's > `HistorizedTagRef` set stays keyed by RawPath (no doubles), and continuous historization / > `EnsureTags` provisioning run once per raw tag regardless of how many equipment reference it. **Equipment-folder event-notifier nodes** serve Event history. Every equipment folder that owns at least one alarm condition is already an event notifier; the server registers a `sourceName` (the equipment id) for each such folder and maps event history reads to the HistorianGateway using that source. (Alarm-history `ReadEvents` requires the gateway running `RuntimeDb:EventReadsEnabled=true`.) Event-field projection supports the standard `BaseEventType` select clauses — `EventId`, `SourceName`, `Time`, `ReceiveTime`, `Message`, and `Severity`; an unsupported select operand returns a null field (spec-conformant). ### No driver-side history (Gitea #655) History is served **only** by the server-wide `IHistorianDataSource` (`GatewayHistorianDataSource` / `NullHistorianDataSource`). No driver contributes history: the OpcUaClient driver does **not** proxy its upstream server's HistoryRead (the former `IHistoryProvider` passthrough was never dispatched to and was deleted). On an OpcUaClient tag, `isHistorized` means "OtOpcUa records this value into the historian from now on", not "forward history reads upstream". A node without `isHistorized` has no HistoryRead access bit and answers `BadHistoryOperationUnsupported`. ### Graceful degradation | Situation | HistoryRead status | |---|---| | Historized node, historian configured and reachable, results found | `Good` | | Historized node, historian configured, time range empty | `GoodNoData` | | Historized node, historian NOT configured (`Enabled=false` / Null source) | `GoodNoData` (empty) for Raw / AtTime; Processed ⇒ `BadAggregateNotSupported` (#682) | | Non-historized node | `BadHistoryOperationUnsupported` | | Backend timeout or exception | `BadHistoryOperationUnsupported` per node; other nodes in the same batch are unaffected | | `Backend=SqlServer`, `AutoCreateSchema=false`, schema missing/behind | every read/write fails with a clear error naming the missing schema version (health `LastError`); reads answer `BadHistoryOperationUnsupported` per node; no host crash | A historized node with no historian configured never returns an error status — it returns empty. This means a deployment can author and publish historized tags before the historian gateway is provisioned, without producing error spikes in connected clients. ### Continuation-point paging (Raw) `HistoryRead-Raw` is paged server-side. The historian backend is single-shot (it returns up to `NumValuesPerNode` samples with no continuation point of its own), so the server synthesises paging time-based: - Every page's backend read asks for **one lookahead row** past the page (`NumValuesPerNode + 1`, plus the boundary ties a resumed page is about to trim — see below). When the lookahead row comes back there is more data: the page returns exactly `NumValuesPerNode` values, and the server stores a resume cursor and returns an opaque continuation point (16 bytes). The client hands it back to fetch the next page. - When it does not, the page is the **last** — no continuation point, even when it is exactly full. (#692: a total that is an exact multiple of the cap used to end with an extra empty `GoodNoData` page.) The exception is an oversized tie cluster (below): its pages are sliced from the over-fetched cluster, its last slice is `min(N, remaining)` values and still carries a continuation point when the window goes on — so stepping past a cluster at the very end of the data can yield one empty final page. - `NumValuesPerNode == 0` ("all values, no limit") is never paged; the whole window returns in one shot. - The resume cursor is **tie-safe**: the next page resumes from the last returned sample's SourceTimestamp *inclusive* and drops the boundary samples already emitted, so samples sharing the boundary timestamp are neither duplicated nor skipped. Because it re-reads them, a resumed page asks for `NumValuesPerNode + skip + 1` rows, so it still carries a full page. On a strict Gateway (`AllowStreamingBeyondCeiling=false`) that moves the row-ceiling edge by `skip + 1`: a `NumValuesPerNode` just under the learned ceiling may now cost one learning refusal (#691.3 re-asks at the ceiling). A page clamp to the learned ceiling is being added separately (#694). - ⚠️ **#692 (client-visible): every page is full** (outside an oversized tie cluster — see above). A resumed page used to ask for only `NumValuesPerNode` rows, so after the trim it netted `N − skip` values — and at `NumValuesPerNode = 1` **nothing**: the lone re-read boundary sample was trimmed, the empty "full" page was taken for a stalled tie cluster, and a `MaxTieClusterOverfetch + 1` (65,537-row) over-fetch stepped one tick to return an **empty page with a continuation point between every value** (live, AVEVA: 8 values = 17 pages / 25 backend reads; on every backend, forward and reverse, with or without `ReturnBounds`). Now 8 values at N = 1 is 8 pages and 8 reads. Clients that drained to the end were unaffected in content, only in cost; a client that stopped at the first empty page lost data. > **Oversized tie clusters — within-timestamp paging.** When more samples share one > `SourceTimestamp` than the current page cap, the server detects that the cursor has stalled on > a tie cluster (every row of the resumed `N + skip + 1` read past the trimmed ties — the lookahead row included — still > sits on the resume timestamp; since #692 nothing else trips it). It then **over-fetches > the entire cluster** at that single timestamp up to a bounded ceiling controlled by > `ServerHistorian:MaxTieClusterOverfetch` (default **65 536**), then serves the cluster > `NumValuesPerNode` samples at a time across successive pages, advancing the cursor one tick past > the timestamp once the cluster is fully drained. A cursor left inside the cluster is marked so (`WithinTieCluster`), and > its next page over-fetches the cluster directly instead of first re-reading rows it knows are ties. A drained cluster > resumes only while the next step is strictly before `EndTime` (truncated to the backend's resolution) — the forward > window is end-exclusive; resuming AT `EndTime` used to make the next read a point read that returned the samples at > `EndTime` as data (#692 review). > > **Reverse reads** (`start > end`, newest first) page the same way in the opposite direction: a drained > cluster steps the cursor one tick **down**, the read ends when that would reach the (excluded) `end`, and > the over-fetched cluster is sliced in descending order. Before the fix a reverse read with a small > `NumValuesPerNode` stopped after its first page with `GoodNoData`. > > **Short pages within a cluster still carry a continuation point.** A within-cluster page that > returns fewer than `NumValuesPerNode` samples (because the cluster happened to be smaller than > the cap, or is the final partial batch) is not the last page if the cluster itself has not been > fully emitted — the server retains the continuation point so the client can drain the remainder. > Only when the cluster is exhausted and the cursor has advanced past the timestamp does the > short-page rule apply. > > **Cluster larger than `MaxTieClusterOverfetch`.** If the over-fetch itself reaches the ceiling > without spanning the full cluster, the node fails **loudly** with > `BadTooManyOperations` (a request limit; `BadHistoryOperationUnsupported` before #686 — events reads changed the same > way) and the tag + timestamp + ceiling are logged. Remedies: raise > `MaxTieClusterOverfetch` (or `NumValuesPerNode`) to cover the full cluster, or investigate the > data anomaly (raw samples normally carry strictly increasing distinct timestamps). > > For a single tag's raw history a tie cluster larger than the default 65 536 is a severe data > anomaly. The ceiling exists to bound server-side memory on pathological data, not to cap normal > operation. **Open-ended and reverse raw reads (#669, #686).** With `NumValuesPerNode > 0` one bound may be unset (Part 11): a start-only read pages **forward** to an open upper bound; an end-only read pages **backward** from `EndTime` — the reverse window `(MinValue, EndTime]`, newest first, so the first page is the `NumValuesPerNode` values at or before `EndTime`. An explicit `StartTime > EndTime` is the reverse window `(EndTime, StartTime]`, newest first. These semantics hold on **every** backend: - **SqlServer / QuestDB** report `IHistorianDataSource.SupportsReverseRaw = true` and are sent the reverse window directly (`… > lo AND … <= hi ORDER BY time DESC`). - **Gateway** (the default backend) reports `false` — the gateway's `ReadRaw` streams ascending only and rejects `start > end` with `InvalidArgument`. The server never sends it one (and the source refuses one unsent, not counted as a health failure); it serves the reverse window from forward reads (`ReverseRawReader`): - **Uncapped** (`NumValuesPerNode = 0`): **one** forward read of the whole window, filtered and reversed in memory — the cost and memory of an uncapped forward read. - **Capped: a bounded walk back.** It reads one segment `(segLo, segHi]` forward, keeps it newest-first, and steps to the next older segment until the page is full or the window is exhausted. A segment is requested as `[⌊segLo⌋, ⌈segHi + 1 tick⌉]`, both rounded **outward to the millisecond**, and filtered on the samples' own timestamps — exact for an inclusive or exclusive forward end, and the row AVEVA fabricates at a query's (possibly rounded-up) start can never land inside the segment. The first segment is **1 minute** (or the width the previous page ended on — it rides the continuation point); each next width is aimed from the density just observed at about half the **page budget** `2N + 16` (≈ N + 8 rows), growing at most ×4. So "the last N values" of a live tag is normally one read of about N rows, a dense tag costs about one N-sized read per page, and sparse history is found in a few dozen tiny ones. - **Read sizes (#689).** A segment read asks for its budget + 2 rows (room for the fabricated row and one to see past the segment). The budget is the page budget `2N + 16`, except a **discovery** segment — the first of a page with no carried width, or one after an empty segment, whose ×4 growth may overshoot into dense data — which may read up to **1,024** rows. The tie budget `M = max(1, MaxTieClusterOverfetch)` is only a **ceiling**: a single timestamp holding more rows than the segment budget escalates to `M`. ⚠️ Before #689 every segment was aimed at half of `M` (≈ 32k rows by default) and that width was carried page to page, so on a 1 Hz tag from about page 5 every page of a "latest N" read fetched ~32,770 rows and took 2–5 s to return N. - **Never answered from a truncated read.** Rows stream ascending, so a read is complete when it came back short or its last row is past the segment; otherwise it is discarded — re-read once with room for the rows before the segment that crowded it (a neighbouring tie cluster, at most twice the budget + 4 rows), or narrowed: to the density it showed, and never again below the newest row it saw until what lies above that row is read. The gap above it is searched: a segment at that density ending on the top (data running right up to it); if empty, a **gallop** up from that row (`(row + d, top]`, `d` ×4 per step, following each truncated read) until a read comes back empty, then a **bisection** of the bracket — never narrower than the 1 ms request granularity, which only re-sent the same request. Measured: a dense tag that stopped two days ago ≈ **26** reads, a burst of just over 1,024 rows that ended 100 days ago ≈ **18** (was 46 bisecting the whole gap) — rather than re-reading the oldest rows of a burst over and over. Only **more than `M` samples within one millisecond** are refused, **`BadTooManyOperations`** (a request limit, not a backend fault); a cluster of exactly `M`, or neighbouring clusters each within `M` and a millisecond or more apart, is served. ⚠️ The unit is the **millisecond**, not the timestamp (#691.4 re-review): requests are rounded to the millisecond, so a narrower segment is the same request — letting the width fall below it once stepped a page through one millisecond a tick at a time (~10,000 reads). Exact for the Gateway, which stores milliseconds; a sub-millisecond backend walked this way would refuse more than `M` samples inside one millisecond even when no single timestamp holds that many. - **Proving the rest empty (#691).** Walking down through empty history to the floor used to cost one read per ×4 growth step — ~14–16 reads (~2.5 s live) on **every** last page of a deep page-through. Now, after two consecutive empty segments, the walk asks **once per page** for the whole remainder down to the floor at the page budget: a complete answer ends the walk in one read; a truncated one (deep history) is discarded and the walk carries on. And the stretch the page's last complete read proved empty below its oldest row (often all the way to the floor) rides the continuation point, so the next page steps straight over it. Data backfilled into such a stretch between pages is not seen by that read — paging was never a snapshot (the forward path misses data inserted behind its cursor too). - **Bounds.** The walk starts no later than **now + 1 day** (a `StartTime = DateTime.MaxValue` read does not send year-9999 query starts) and never reads before the **Unix epoch** (AVEVA holds nothing earlier): a walked-back read returns only samples strictly after 1970-01-01T00:00:00Z and no later than now + 1 day. Paging is otherwise unchanged — a reverse cursor resumes as a fresh walk from its resume point, carrying the width and the proven-empty stretch (session-held cursor state, never serialized, so the new fields need no versioning). - **Null** reports `true` (empty in either direction). > ⚠️ **Client-visible (#686).** Since #669 an end-only capped raw read, and any explicit reverse raw read, on the Gateway > backend failed the node (`BadHistoryOperationUnsupported`) and was counted as a historian backend failure (the gateway > refused `start > end`). Before #669 an end-only read on Gateway returned the **oldest** N values ascending. Both now return > the Part 11 answer — the latest values at or before `EndTime`, newest first. **Events** follow since #691.4 (see below). > > ⚠️ **Also fixed with #686: uncapped raw reads on Gateway.** `NumValuesPerNode = 0` ("all values") used to be forwarded as > `max_values = 0`, which the client package and the gateway reject — every uncapped raw read (forward too) failed the node > and counted as a backend failure. It is now sent unbounded (`int.MaxValue`); a gateway configured with > `AllowStreamingBeyondCeiling=false` refuses that over its `MaxValuesCeiling`; since #691.3 the read is retried once at the > ceiling the refusal names and answered when provably complete, else **`BadTooManyOperations`** (not a backend fault) — see > [#687](#gateway-raw-reads-return-stored-samples-only-687) "Strict gateway". Since #694 the HistoryRead pages such a read at the > learned ceiling with continuation points instead (same section, "Pages are sized to a strict gateway's ceiling"). A cancelled read (client cancel / `HistoryReadDeadline`) is no longer counted as a backend failure. Continuation points are bound to the OPC UA session (the SDK's `ServerConfiguration.MaxHistoryContinuationPoints` cap, default 100, with oldest-eviction; points are disposed when the session closes). Resuming an unknown / evicted / released point returns `BadContinuationPointInvalid`. `releaseContinuationPoints` drops the stored cursors without reading data. ### Gateway raw reads return stored samples only (#687) > ⚠️ **Client-visible (#687).** An AVEVA `Full` raw read starts with a synthetic row stamped exactly at the **query start**: the > value in effect there (AVEVA `Quality = 133`, the previous sample's value and quality — a value never stored at that time) or, > before the tag's first sample, a NULL row (value 0, Bad, `Quality = 1`). The `Gateway` backend used to return that row as a stored > sample: a window with no stored sample in it answered one fabricated value stamped at StartTime (live: `[18:06:21, 18:06:39]` ⇒ > `18:06:21 v=30 Good`), a window before the tag's history answered `v=0 Bad` instead of `GoodNoData`, and a start-only read led > with the fabricated row. It is now dropped; such windows read `GoodNoData`. - **How.** The same request start the raw-evaluated aggregates use (#681, `RawAggregateReader.RequestStartFor` — one authority): the gateway is asked from **1 ms before** the window (never before the Unix epoch, the start of the historian's time encoding), so the boundary row lands before it, and only rows `RawAggregateReader.IsWindowSample` are returned. The decision is by **timestamp**, not by value or quality — a stored NULL sample looks exactly like the no-data boundary row on the wire (value 0, Bad), and a stored NULL inside the window is still returned (Bad), as it is a stored sample. ⚠️ **Since #690 it is returned with a null value** (client-visible): histsdk puts a value on every row, so the NULL used to come back as value `0` + Bad (`""` on a string tag), a fabricated 0 for any client that reads the value without the status. `SampleMapper.ToSnapshot` drops the value of every row whose mapped status is **Bad** (AVEVA stores a Bad-quality sample as NULL, so a Bad row never carries a real value) — raw reads and AtTime alike, matching SqlServer/QuestDB (an Empty sample reads null + Bad). Uncertain rows keep their value; the raw-evaluated aggregates were already unaffected (a Bad sample is never usable, and Start/End null a Bad value). A row stamped **at or before the epoch** is never a stored sample (AVEVA stores nothing there — the #686 walk-back's floor too): a window starting at or before the epoch gets no margin, so AVEVA's no-data row lands exactly at its start; raw reads and the per-sample aggregates both drop it. - **The forward end is exclusive** (`[StartTime, EndTime)`, the `IHistorianDataSource` contract and what the SQL backends return; the gateway's own `ReadRaw` end is inclusive, so a sample stored exactly at EndTime used to be returned). A point read (`StartTime == EndTime` — the server's tie-cluster over-fetch) still returns the samples at that instant. The #686 reverse walk-back is exact either way. - **Row budget.** `NumValuesPerNode = 0` ("all values") asks the gateway for an unbounded `max_values` (`int.MaxValue` — the client package and the gateway reject 0, #686). A cap asks for `NumValuesPerNode + 16` rows, **not** clamped at the gateway's default ceiling (#688 clamps only the per-sample aggregate budget): AVEVA always emits the boundary row, so a clamped budget would make every page size from 999,985 to 1,000,000 come back short. A default gateway serves any budget. - ⚠️ **Strict gateway (#691.3, client-visible).** A gateway with `AllowStreamingBeyondCeiling=false` refuses a budget over its `MaxValuesCeiling` (before streaming) and names the ceiling in the refusal; the gateway has no RPC that reports it. The read is then **re-issued once with the ceiling as the budget**, and the ceiling is **remembered per data source** (`GatewayRowCeiling`, shared by raw and per-sample reads): every refused attempt counts as an error in the gateway's own outcome metrics, so after the first refusal later reads ask for at most the learned ceiling and are not refused. A **lowered** ceiling is re-learned from the refusal of the remembered one. A **raised** ceiling (or a switch to streaming) is noticed by the only reads it matters to: when a read bounded by the remembered ceiling comes back full without proving completeness, it is re-probed **once** at its full budget — served ⇒ the ceiling is forgotten; refused naming a larger ceiling ⇒ that one is learned and the read re-issued at it; refused naming the same ceiling ⇒ the bounded result stands. No timer: a TTL would only add refusals for reads that already fit; the cost of the rule is one refused re-probe per genuinely over-ceiling read (which fails either way). So an uncapped read (`NumValuesPerNode = 0`, which used to be refused outright on a strict gateway, even for a 10-row window) and a page size within 16 of the ceiling are now served when the window fits. A ceiling-bounded read is answered only when it is **provably complete** — the stream came back short of the ceiling, or reached `EndTime` (or, capped, filled the page); a stream that delivered the whole ceiling short of `EndTime` may be truncated and fails **`BadTooManyOperations`** (not a backend fault) — set `NumValuesPerNode` to page, or narrow the window. A refusal whose text carries no number falls back to the gateway's default ceiling (1,000,000). Every stream is still drained. - ⚠️ **Pages are sized to a strict gateway's ceiling (#694, client-visible).** Until #694 a page size above a strict gateway's ceiling failed: with `MaxValuesCeiling = 1000`, `NumValuesPerNode = 5000` — or `0` over 3,600 rows — was `BadTooManyOperations`. The data source now reports the learned ceiling to the server as `IHistorianDataSource.MaxRawValuesPerRead` = **L** = ceiling − 16 (`0` until a refusal has named a ceiling, and again after a re-probe finds the gateway streaming; `0` on every SQL backend). The node manager plans each page with it (`RawPageLimit`): a page's read is the page **P** + the boundary ties **k** it re-reads and trims + the #692 lookahead row, and with a limit that read is at most L — so **P = min(N, L − k − 1)** (uncapped: P = L − k − 1; with the ceiling at 1000 a first page holds 983 values, a resumed one 982), and a full page carries a continuation point like any other (Part 11 allows fewer than `NumValuesPerNode` with one). **This includes uncapped reads** (`NumValuesPerNode = 0`), which page instead of being refused; a window that fits is still one page without a continuation point. Without a limit the #692 plan is unchanged. Boundary ties that alone fill a read (k ≥ L − 1) go straight to the tie-cluster over-fetch — never an uncapped or underflowing request. - **First read after start-up:** the ceiling is unknown until refused once; the read that learned it is re-planned and re-issued ONCE (cost on that read: the refused attempt, the ceiling re-read that could not hold the window, the page-sized read; later pages are one read each). Reverse / end-only reads (the #686 walk-back) size their segments to L (tie budget `min(MaxTieClusterOverfetch, L − 2)`); on the page path the refusal reaches the page planner, which shrinks the **page** — the walk never re-runs itself at the ceiling with the page still at N (review F2: a cold end-only read with a ceiling of 20 and N = 3000 cost 3,023 gateway reads in one HistoryRead; now ≤ 40). Bound lookups (a page of one) keep the walk's own re-run. - **Crowded look-back (review F3).** A page asking for exactly the ceiling leaves no room for the #687 1 ms look-back, so more than 16 rows just before a page start (e.g. 20 ties 1 ms before it) used to fail `BadTooManyOperations`. The data source now reports the crowding rows on its refusal (`HistoryReadLimitExceededException.PreWindowRows`, only when they exceed the 16-row allowance), and the planner re-reads the page once that many values smaller — the source's own crowding re-read then fits under the ceiling. In the walk-back, a crowded segment is re-read that much smaller and counts as truncated. - **Tie clusters.** The over-fetch that pages within a cluster reads at most `min(MaxTieClusterOverfetch, L − 1)` + 1 rows (so it never trips a refusal); a larger cluster is **`BadTooManyOperations`** — the gateway cannot offset into it. A crowding refusal that leaves no room for a page is that answer too (it is never mistaken for "the cursor sits in a cluster", which would step the cursor a tick per page forever). Every sample before such a failure is served exactly once; nothing is truncated silently. - **Never a truncation:** the gateway read is still answered in full or refused; only the server's page is smaller. - A raised ceiling is not noticed by raw paging (its reads ask for at most the learned ceiling, so nothing re-probes); pages stay at the old size until a read asking for more (a per-sample aggregate) re-probes it, or the server restarts — correct results either way. A **lowered** ceiling is re-learned from the next page's refusal and that page re-planned. - ⚠️ **A window starting at or before the Unix epoch (review F1, fixed with #694).** Such a window gets no 1 ms look-back, so AVEVA's synthetic row lands exactly at its start; it is no sample, and it used to count as "past the window end" — a ceiling-truncated read was then reported complete (an uncapped forward read from the epoch answered its first ceiling − 1 rows, Good, no continuation point). It now counts as a pre-window row. - **Paging (#669) is unaffected.** A full page still holds exactly the page size of stored samples and carries a continuation point, and a resumed page (which re-reads from the last returned timestamp inclusive, and the tie-cluster over-fetch `start == end` read) no longer gets a fabricated row at its resume point. A read is complete when the stream came back short, filled the page, or reached EndTime. If rows **before** the window — the boundary row plus stored samples inside the 1 ms margin, e.g. a tie cluster at `StartTime − 1 ms` or at a resume point `− 1 ms` — used up the budget, the window is re-read with room for them: `cap + room + 16`, room = the pre-window rows the first read saw, at least **doubling** per re-read while a read that was itself cut short keeps finding more (up to 8 re-reads, never past a known strict ceiling — review F4: a single re-read sized from a truncated read's count failed a 300–1,700-row cluster just before a resume point even on a default gateway). Only if the re-reads are still crowded does the node fail `BadTooManyOperations` (not a backend fault) rather than answer a short page that would end the client's paging early. Every gateway stream is still drained to completion. - **`ReturnBounds` is served by the server, not by this row** (#691, [Bounding values](#bounding-values-returnbounds-691)). The AVEVA boundary row is never used as a bound: it is stamped at the query start rather than carrying the time of the real prior sample, so it could only ever be a Part 11 *interpolated* bound. The start bound on the Gateway is the real prior sample, found by the #686 walk-back. ### Gateway AtTime is evaluated from raw reads (#693) > ⚠️ **Client-visible (#693).** Gateway AtTime used to pass AVEVA's own interpolated answer (the gateway's `ReadAtTime`) through, > which broke the [AtTime rules](#attime-reads) every other backend follows (Part 11 §6.5.4). Live (`HistGW.LiveTest.Gate685`: > 100 @ 19:00:00, 107 @ 19:00:02, **NULL** @ 19:00:03, 108 @ 19:00:05) it returned **107 Good at the NULL's own time**, **null + Bad > at 19:00:05** (a stored Good 108 right after the NULL), **99.99999999944 for the stored 100**, and no Interpolated bit on any > interpolated value. It is now answered from **raw** reads with the SQL backends' shared `AtTimeEvaluator` (as #681 did for the > per-sample aggregates), so the same stored samples give the same answer on every backend: the exact stored value (no > Interpolated bit) at a stored time; null + the outage's Bad code at or after a stored NULL; the previous value held, > `UncertainDataSubNormal` + Interpolated, before a Bad neighbour; linear + Interpolated between usable neighbours; `BadNoData` > before the first sample; stepped after the last (`UncertainDataSubNormal` once later than now). Kill switch: > `ServerHistorian:GatewayAtTimeFromRaw = false`. - **Read plan (`GatewayAtTimeReader`).** Each requested time needs its stored sample at-or-before (`prev`) and after (`next`; read only when the evaluator uses it — a usable `prev` that is not an exact hit). The distinct times are served by **one** forward window read `[first − 1 ms, last + 1 ms]` of at most 1,024 rows, plus a **walk-back** for the sample before the first time and one small forward **probe** (18 rows) for the sample after the last. Measured on the fake: ten times one second apart on a 1 Hz tag cost **1 read** (all exact hits) or **3** (interpolated); ten times an hour apart on a 1 Hz tag **12** (≈ 9.5k rows, never the 32k between); one time 400 days after the only samples **5**; a tag with no history **4**. A window read that fills its budget resolves the times before its last row; the next time either continues from there (if the rows in between should fit one read) or **jumps** to it with a small read (64 rows) that starts a density-sized margin earlier (so its `prev` is normally in the same read) — 1,000 times spread over ten years of a 1 Hz tag cost ≈ 1,000 reads of ≤ 64 rows (≈ 66k rows; it used to be 1,024-row reads). - **Walk-back** (the gateway reads forward only), in two phases. **Gallop:** segments stepping back from the time, the first sized from the window read's density (else 1 minute), growing ×4 when empty; after two empty segments the whole remainder down to the epoch is asked once — a complete answer ends the walk, a **full** one seeds the bisection with its last row. **Bisect** (once a stored row is known below — a full segment's, or an earlier window read's last row): read `(mid, hi]`; empty ⇒ `hi = mid`; full ⇒ the known row moves up to its last row; complete with rows ⇒ done. Right after a full read one segment is aimed at about a quarter of a read at the density it showed (until an aimed segment comes back empty — a gap — then plain halving); below 2 ms the segment is the whole stretch from the known row (never a 100 ns step). Measured (review harness, 5 years of history, time = now − 1 min): a tag stopped 1 h / 1 d / 7 d / 30 d / 365 d ago costs 6–18 reads at 1/min, 21–25 at 1 Hz and 25–31 at 10 Hz; a 1 kHz tag behind a 10-minute gap 6 reads (< 10k rows). (Before the review fix a full segment reset the width and a full whole-remainder read was discarded, so a dense tag that stopped weeks ago gave up after 65 reads.) More than 1,024 stored samples within one millisecond fail the node `BadTooManyOperations` (not counted as a backend failure). - **Bounded.** One request takes at most **64 + 16 × distinct times** raw reads; past it the node fails `BadTooManyOperations` ("needed more than N raw reads", naming a small strict-gateway ceiling when one bounded the reads). It is a runaway guard, not a budget (`HistoryReadDeadline` bounds the wall clock): a time inside a gap of a bursty tag legitimately costs a jump read plus a bisection back across the gap, ≈ 6 reads — a 1 h/day 1 Hz tag asked at noon on 30 / 100 / 300 consecutive days takes 186 / 606 / 1,806 reads, which the first cut's 64 + 4 per time refused although AVEVA's own AtTime served it. A strict gateway whose ceiling is below 4 rows is refused up front with a message naming the ceiling; a small one (e.g. 10) is served by paging within the cap. - **Wire shape.** Every request bound is a whole millisecond and a row counts only **strictly inside** its request, so AVEVA's synthetic boundary row at the query start (#687) — and any row at the query end — is never taken for a stored sample. A Bad row is value-less (#690). Nothing is asked past now + 1 day (the gateway's AVEVA may refuse a far-future end; a time beyond it is stepped from the last sample). Every stream is drained. A strict gateway's ceiling (#691.3) is learned on its first refusal and honoured; a full read is ordinary paging here, so it is not re-probed. - **Values follow the node's type (type hint).** The gateway wire carries no data type — every historizable type is stored as an AVEVA analog (a Boolean as a 0/1 `Int2`) — so the node manager passes each node's scalar DataType with the AtTime call (`IHistorianDataSource.ReadAtTimeAsync(…, TypeCode? valueType, …)`, a default-interface overload: SqlServer / QuestDB / Null ignore it; Raw and UNS NodeIds give the same hint, arrays and other types none). The Gateway converts each raw row to that type **before** the shared evaluator runs, so its own type rules apply exactly as on the SQL backends: a **Boolean** node (value ≠ 0) is **stepped** — never 0.5; an **integer** node is interpolated, rounded half away from zero and returned as the node's own integer type (`Int32` ⇒ `int`); Float / Double interpolate as Double (as the SQL backends return them). With no hint (an unknown node) every numeric tag interpolates as Double. A **DateTime** node is stepped, but its value stays the gateway's Double: the Gateway never historizes a DateTime node (`HistorianTypeMapper` skips it at provisioning and the recorder drops non-numeric values), so such a node only has history if something else wrote that AVEVA tag, and the numeric encoding is not known here — the SQL backends store DateTime as its own kind and return a `DateTime`. Under an integer hint a NaN, ±∞ or out-of-Int64-range stored value is no integer: it is treated as unusable (as `BadNoData`), so a neighbour like that is never interpolated or saturated into a Good value. - **What differs between backends (by design, not changed by #693).** Only the **Gateway** converts an AtTime result to the node's integer type (`Int32` ⇒ `int`): the SQL backends store and return integer nodes as `Int64` (and Boolean as Boolean). Gateway **Raw** reads still return a Boolean node's samples as the stored Double 0/1 (the hint is passed only on AtTime). - ⚠️ **Shared-evaluator fix (all backends that use it):** `AtTimeEvaluator` returned every interpolated `Int64` boxed as a **Double** (a C# `cond ? (long)x : double` conditional widens the long); its "keeping the Int64 type" tests passed because Shouldly's `ShouldBe` compares numbers across types. SqlServer / QuestDB interpolated `Int64` AtTime values are now `Int64`, as documented in [AtTime reads](#attime-reads). ### Bounding values (`ReturnBounds`, #691) ⚠️ **Client-visible — a breaking change for .NET clients that rely on SDK defaults.** Until #691 `ReadRawModifiedDetails.ReturnBounds` was ignored on every backend. It is now honoured on **every** backend (SqlServer, QuestDB, Gateway, and none), following OPC UA Part 11 §4.6 / §6.5.3.2 and checked against every row of Part 11 Table 1. **The OPC Foundation .NET SDK's `new ReadRawModifiedDetails()` defaults BOTH `ReturnBounds = true` AND `IsReadModified = true`** (verified on 1.5.378.106), so: - a client that clears `IsReadModified` but never sets `ReturnBounds` used to get stored values only and **now gets bounding values**; - a client that sets neither (a bare `new ReadRawModifiedDetails { StartTime, EndTime }`) is a *modified* read with bounds, which used to be served (or `BadHistoryOperationUnsupported` on a backend with no trail) and is **now `Bad_InvalidArgument`** (§6.5.3.3). Set both explicitly. `Client.Shared` (Client.CLI / Client.UI) already does, and ScadaBridge does not issue HistoryRead. The kill switch `ServerHistorian:HonorReturnBounds = false` restores the pre-#691 behaviour (both changes) fleet-wide. - **What a bound is.** Forward (`StartTime < EndTime`): the start bound is the stored sample **at or immediately before** StartTime, the end bound the one **at or immediately after** EndTime. Reverse mirrors it (start bound at or after StartTime, end bound at or before EndTime), newest first. A sample exactly **on** StartTime is the bound and is already the data's first value (nothing extra is added); a sample exactly on EndTime is added as the end bound (the data window excludes EndTime). `StartTime == EndTime` reads forward: the data is the samples at that instant, the end bound the first sample strictly after it. - **A bound is a stored sample of any quality** — a Bad sample or a comms-loss outage marker comes back as stored. Inside a tie cluster the bound is the cluster's first sample in the read's direction. - **Nothing at all ⇒ `GoodNoData`.** A read with no data in the window **and** neither bound found returns no values and `GoodNoData` — the answer it gave before bounds (§6.5.3.2: bounds make the result a success only "if one or both exist"). This is judged over the whole read (it only arises on a single, final first page); once any bound or value exists, a missing bound is returned as its placeholder. - **Missing bound:** `Bad_BoundNotFound`, null value, stamped with StartTime / EndTime. A one-sided read's open side (only a StartTime, or only an EndTime) is never looked up; it ends the read as `Bad_BoundNotFound` stamped one second after (forward) or before (reverse) the previous returned value (Table 1 footnotes a/b). `Bad_BoundNotSupported` is never returned — every backend can find a bound. - **Counting and paging.** Bounds **count toward `NumValuesPerNode`** (§6.5.3.2: "any Bounding Values returned are included in the numValuesPerNode count"; `NumValuesPerNode = 1` returns the start bound alone). The start bound leads the **first** page and the end bound ends the **last**: the pending end bound rides the continuation point, and when a page is exactly full the end bound is the whole next page. Paging a bounded read at any page size concatenates to the unpaged answer. At `NumValuesPerNode = 1` with no start bound found, the server first checks the read holds anything (else `GoodNoData`); since #692 what that check learns rides the continuation point — the walk-back hint and, when there is no data, the end bound it found — so the next page reads nothing it already read. - **Placeholders are marked, not recognised by shape (#692).** Only a `Bad_BoundNotFound` value the server synthesised is a placeholder; a stored row that carries `Bad_BoundNotFound` with a null value (writable through HistoryUpdate) is data, and never turns a read into the "no data and no bound" `GoodNoData` answer. - **How it is served — one authority, `HistoryBounds` in the OPC UA server.** "At or before T" is a newest-first read of `(0, T]` capped at one — native on SqlServer/QuestDB, the #686 walk-back of bounded forward reads on the Gateway (a tag with no sample before StartTime is proven empty down to 1970 by the #689 walk — after two empty segments it asks once for the whole remainder — so a missing start bound costs a few small reads, not one per ×4 step). "At or after T" is a forward read of `[T, MaxValue)` capped at one — on the **Gateway** it ends at **now + 1 day** instead (the walk-back's horizon; the HistorianGateway only checks `start ≤ end`, so a year-9999 end would reach AVEVA), and an end bound beyond that horizon is `Bad_BoundNotFound` without a read. SqlServer/QuestDB keep `MaxValue` (they bind it safely and may hold samples further ahead). "Strictly after" starts one storage resolution later. On QuestDB, whose window bounds truncate to the microsecond, "on the bound" compares at that resolution, so a sample at ⌊StartTime⌋ is never returned twice. A first page therefore costs up to three backend reads (start bound, data, end bound). The `BoundingValueConformance` suite pins those exact lookup reads on SqlServer and QuestDB. ⚠️ On the **Gateway** the start-bound walk-back is subject to the #689 per-millisecond tie limit: if the millisecond holding the bound has more than `MaxTieClusterOverfetch` samples, the lookup cannot pick the cluster's last sample without reading it all, so the **whole node** read fails `BadTooManyOperations` (even though the data read alone would succeed) rather than return a wrong bound. - **Modified reads:** `IsReadModified = true` with `ReturnBounds = true` is `Bad_InvalidArgument` per node (Part 11 §6.5.3.3). - **Kill switch:** `ServerHistorian:HonorReturnBounds` (default `true`). `false` ignores `ReturnBounds` exactly as before #691 — stored samples only, and no modified-read rejection. - **Not affected:** Processed, AtTime and Events reads (they have no `ReturnBounds`). ### Continuation-point paging (Events) (#669) A capped event read (`NumValuesPerNode > 0`) pages exactly like Raw, with one difference: the trigger is the **source's truncation flag** (`HistoricalEventsResult.ContinuationPoint != null`), never the page count, so a cap-sized page the source says is complete carries no continuation point. `NumValuesPerNode == 0` stays unbounded and never pages. - The cursor is `(source, next start, end, boundary skip, cap)` in the same session-bound store as Raw (flagged as an events cursor — a Raw point presented to an events read, or the reverse, is `BadContinuationPointInvalid`). The next page reads from the last event's `EventTimeUtc` **inclusive** and drops the events at that timestamp already emitted — and asks the source for that many more rows than the cap, so every page short of the last is a FULL `NumValuesPerNode` page (#675; resumed pages used to come back short by the skip). - A tie cluster larger than the cap is paged within by over-fetching the half-open window `[T, T + resolution)` up to `MaxTieClusterOverfetch`; beyond that bound the node answers `BadTooManyOperations` (raise the bound). - **Tie exactness depends on the backend.** SqlServer and QuestDB order by `(EventTimeUtc, EventId)`, so the skip count is exact. The Gateway backend has no stable intra-timestamp order (and only a truncation flag), so ties at a page boundary of a **forward** read are best-effort there; a walked-back **reverse** read (below) sorts them by `EventId` itself and is exact. - **Reverse event reads page newest-first (#675).** `StartTime > EndTime` reads the Part 11 reverse window `(EndTime, StartTime]` with events in DESCENDING `(EventTimeUtc, EventId)` order, so a cap keeps the **latest** N and a continuation point is issued when the source flags more. The direction is stored **in the cursor**: a reverse point resumes reverse (and a forward one forward) whatever the resume request's details say. Reverse tie clusters are over-fetched forward and sliced from the newest end, exactly as Raw does. Before #675 a capped reverse read silently returned the EARLIEST N, ascending, Good, with no continuation point. - **End-only events reads page backward on every backend (#675, #691.4).** `StartTime` unset (`DateTime.MinValue`) with an `EndTime` and `NumValuesPerNode > 0` pages backward from `EndTime` over `(MinValue, EndTime]` — the same rule as Raw. Start-only reads page forward to an open upper bound (`DateTime.MaxValue`, UTC) everywhere. Without a cap an end-only read is forward `[MinValue, EndTime)` everywhere. - **Backends.** The server checks `IHistorianDataSource.SupportsReverseEvents`. SqlServer and QuestDB serve reverse natively (`… > lo AND … <= hi ORDER BY time DESC, id DESC`, the `ReadEventsAsync(…, bool reverse, …)` overload) and report `true`. **Gateway reports `false`** — the gateway streams ascending only and rejects `end < start` — and is never sent a reverse read (Null reports `true` and answers empty either way, as for raw): since **#691.4** the server walks it back from bounded **forward** reads with the raw walk (#686/#689/#691: page-sized ms-aligned segments filtered to `(segLo, segHi]`, a read the source flagged truncated is never answered from, the now + 1 day / 1970 bounds, the remainder probe and the proven-empty stretch carried in the cursor). Event specifics: - **Ties** are sorted by `EventId` (ordinal) on every read — the gateway orders by `EventTimeUtc` only, in any order within a timestamp — so the page is DESCENDING `(EventTimeUtc, EventId)` and a resumed page's boundary skip lines up; the oversized-cluster over-fetch keeps exactly the events at `T` (the gateway's end is inclusive) in the same order. - **"More exists"** is found by walking for one event past the page (the walk has no source truncation flag to hand on). - **Uncapped** reverse reads are one forward read of the window, sorted and reversed. - ⚠️ **Also fixed (#691.4 review): uncapped events reads on Gateway.** `NumValuesPerNode = 0` reaches the source as `int.MaxValue`, and `GatewayHistorianDataSource` pre-sized its result list to it ("Array dimensions exceeded") — every uncapped Gateway events read, forward too, failed the node and counted as a backend failure. The capacity is now bounded. - A timestamp holding more than `MaxTieClusterOverfetch` events ⇒ **`BadTooManyOperations`** for that node. - ⚠️ **Gateway prerequisite:** the gateway's `RuntimeDb:EventReadMaxRows` must be `0` (its default, unlimited) or larger than the walk's biggest read (`2N + 18` rows, a discovery read 1,026, a tie-cluster read up to `2·MaxTieClusterOverfetch + 4`). That cap truncates **silently** (the stream just ends), so a segment it cut would look complete and drop its newest events — the same blindness a forward read has to it, but on the end a reverse read is asking for. - ⚠️ **Cost of "nothing older" (#694 item 2).** A walked read that runs out of events proves the rest of its window empty with one forward read down to its earlier bound — for an end-only read, the Unix epoch. On `Runtime.dbo.Events` that read is measured at **~3.8 s** for a source with few events (vs ~0.7 s for a bounded window), and an **unfiltered** deep read hit the gateway's 30 s SQL timeout; a forward read from a far-past `StartTime` scans the same range. The cost is the gateway's query over `Runtime.dbo.Events` (its index use is a historiangw matter, not fixable here). Two ways to keep it down: give the read an explicit `EndTime`/`StartTime` that bounds the window, or set the opt-in **`ServerHistorian:EventReadHorizon`** (below). The default stays exact — there is no cache of "earliest event per source", deliberately: events reach `Runtime.dbo.Events` from the alarm store-and-forward drain (which can deliver hours- or days-old events after an outage) and from AVEVA itself, so a remembered "nothing before X" can go stale silently. ⚠️ **Client-visible (#691.4).** On Gateway (the default backend) and Null an explicit reverse events read used to be refused `BadHistoryOperationUnsupported` (#675), and an end-only capped events read returned the **oldest** N events ascending. Both now return the latest events newest-first. Because the capability is checked, not a caught exception, a `NotSupportedException` thrown from inside a backend (a driver library) is still logged as a backend fault. Every implementation (and any future wrapper) must declare both members itself — the interface defaults (`false` / throw) would silently send a native-reverse backend through the walk; `HistorianDataSourceReverseEventsGuardTests` pins it. - **Gateway open upper bound.** The `DateTime.MaxValue` (UTC) bound of a start-only forward read is pinned offline through the published client's proto `Timestamp` conversion (captured by an in-process gRPC server) and the FILETIME encoding (`GatewayEventBoundsTests`). What a live AVEVA Historian does with a year-9999 bound is still unverified (needs the VPN). - No session on the request (in-process callers) degrades to single-shot, as for Raw. ### Modified-value history (Part 11) **Gitea #668.** On the `SqlServer` and `QuestDb` backends every HistoryUpdate leaves a **modification trail**: one row per sample it inserted, replaced or deleted, carrying the value **as it was before** the change (none for an insert), the caller's principal, the modification time and the Part 11 `HistoryUpdateType`. A HistoryReadRaw with `IsReadModified = true` returns that trail as a `HistoryModifiedData` — one `DataValue` (the prior value) plus one parallel `ModificationInfo` (`ModificationTime`, `UpdateType`, `UserName`) per entry. | Operation | Recorded `UpdateType` | Prior value | |---|---|---| | Insert | `Insert` | none — served `null` / `BadNoData` | | Replace | `Replace` | the replaced value and its status | | Update (upsert) | `Update` | the replaced value, or none when it inserted | | DeleteAtTime / DeleteRaw | `Delete` | the deleted value and its status | - **Only applied items are recorded.** A refused item (`BadEntryExists`, `BadNoEntryExists`, `BadInvalidTimestamp`, …), a DeleteRaw over an empty range, the **live recorder's writes** and the **retention purge** leave nothing. - **`UserName` is the session's principal** — the API key's `NodeAcl` principal, or `anonymous` for an admitted anonymous session (the key id is stored too, not served). Longer than 200 characters is truncated, never refused. - **Window and order.** Same windows as Raw: forward `[start, end)` ascending by `(SourceTime, ModifiedId)`, reverse (`start > end`) `(end, start]` descending, `start == end` the modifications at exactly that time; an unset bound with a cap pages open-ended like Raw. Several modifications of one timestamp appear in the order they were made. - **Paging.** `NumValuesPerNode > 0` pages with session-bound continuation points keyed on the last entry's `(SourceTime, ModifiedId)` — an exact, exclusive key, so there are no ties to over-fetch and no trailing empty page (the backend reads `cap + 1` rows to know). The cursor is bound to the node and to the modified read kind (a raw or events point is `BadContinuationPointInvalid`). `NumValuesPerNode == 0` is unbounded. - **Authorization** is the ordinary **HistoryRead** decision — no new operation or permission (ungated with `Security:Acl:Enabled` off). - **Unsupported** ⇒ `BadHistoryOperationUnsupported`: the `Gateway` backend (the AVEVA Historian exposes no modification trail) and the `Null` default. **Deleting the trail** (`DeleteRawModifiedDetails.IsDeleteModified = true`) is not supported either — the trail is an audit record and is aged out only by retention (below). - **Atomicity differs by backend.** `SqlServer` writes the trail row with `OUTPUT … INTO` in the **same statement** that changes `hist.Sample`, so a change and its trail row commit or roll back together. QuestDB has no multi-table transaction, so the trail row is written **before** the change: a crash (or failed sample INSERT) in between leaves an **orphan trail row** describing a change that never landed — never a change without its trail row. Treat a QuestDB trail entry whose sample still shows the prior value as such an orphan. A failed sample write after its trail rows landed logs a **Warning** naming the tag, the row count and the time range, so the orphan can be found. - **Rolling upgrade (SqlServer).** The v2 schema is applied by the first node that upgrades. Until its peer is upgraded too, that not-yet-upgraded node still runs the old build, whose HistoryUpdates write **no trail rows** — silent gaps in the audit trail. Upgrade both nodes of a pair together (stop, upgrade, start both); modifications made by an old-build node in the window are never recorded. - **Paging vs a concurrent HistoryUpdate (SqlServer).** `ModifiedId` is an `IDENTITY`, which orders by allocation, not by commit. A modified read paging at the same moment a HistoryUpdate commits a modification of the **same source timestamp** can have already passed that row's id and so never see it on this pass (a fresh read does). The cursor is exact for a quiescent trail, not a snapshot guarantee. - **QuestDB `ModifiedId` across nodes.** Ids are UTC ticks bumped past the last one issued by that process, so a pair's ids interleave by each node's **wall clock**. Same-source-time modifications made on two nodes order by their clocks (clock skew can reorder them); within one node the order is strict. - **Retention.** The trail ages out on the same **source-time** cutoff as the samples it describes: `SqlServer` purges `hist.SampleModified` per tag alongside `hist.Sample`; QuestDB puts the same TTL on `{p}sample_modified`. - **SqlServer schema v1 (#674).** With `AutoCreateSchema=false` a database still at schema **version 1** (script `002_sample_modified.sql` not applied) keeps historizing — live writes, raw/processed/at-time reads, alarm events and retention all work — but has no trail: modified reads and every HistoryUpdate request answer a **node-level `BadHistoryOperationUnsupported`** (all three arms alike — the updater throws `NotSupportedException`, as the read side does) with nothing sent to SQL, retention skips `SampleModified`, and the node logs **one Warning per boot**. ⚠️ Historized nodes still advertise the `HistoryWrite` AccessLevel bit on a v1 database: the bit is decided at materialise time from "an updater is wired", and whether the schema has the trail is only known after the backend's lazy first-use bootstrap (I/O the node manager does not do while materialising), so a v1 node is admitted to HistoryUpdate and then refused. Apply `002_sample_modified.sql` and restart the node to enable them (the verdict is taken once per process). `AutoCreateSchema=true` (the default) still upgrades v1 to v2 in place. See [Schema](#schema). - **QuestDB retry duplicates (#674).** `{p}sample_modified` is append-only with **no DEDUP key** (a key would collapse genuinely distinct modifications of one sample), and each trail row gets a fresh `modified_id`. So a HistoryUpdate a client **retries** after a timeout — when the first attempt had in fact landed — records a second trail row for the same change (and, for Insert, the retry is `BadEntryExists`). The sample itself is idempotent (dedup upsert); only the audit trail can show the duplicate. `SqlServer` has the same exposure only if the client retries a committed request; there is no request id to dedupe on in either backend. - **Cross-node ordering (QuestDB).** The per-tag lock is per process; the [Primary-only](#primary-only-674) HistoryUpdate gate (#674) is what keeps two nodes of a pair from recording concurrent updates of one tag (and so a wrong prior value). Two **separate** pairs sharing one QuestDB can still race; that is unsupported. - ⚠️ **Who can read the trail (#674.4).** Modified reads are ordinary HistoryReads: with `Security:Acl:Enabled=false` (the default) **any admitted session, including Anonymous, can read who rewrote history** (`UserName` is the principal). Turn the ACL flag on, or set `OpcUa:AllowAnonymous=false`, if the principal names are sensitive (see [security.md](security.md)). - **Not implemented (#674.5):** `IsDeleteModified` (deleting trail entries — `BadHistoryOperationUnsupported`); recording an **Update** (upsert) as a split `Insert` / `Replace` — it is recorded as `Update` either way; and **`AuditHistoryUpdateEvent`** (no audit event is emitted for a history change — the trail is the only record). ```bash otopcua-cli historyread -u opc.tcp://localhost:4840 -U apikey -P otua__ \ -n "ns=2;s=Plant/Modbus/dev1/Speed" --start 2026-01-15T00:00:00Z --end 2026-01-16T00:00:00Z --modified ``` ### Total aggregate The OPC UA `Total` aggregate is **supported** over the HistorianGateway backend. The gateway exposes a native **`Integral`** retrieval mode, so `Total` maps straight to it (`HistoryAggregateType.Total → RetrievalMode.Integral`) — no client-side scaling. (This replaces the retired Wonderware path, which had no `Total` column and derived it client-side as time-weighted `Average × interval-seconds`.) Values pass through unchanged; a Good bucket is `Good | Calculated` (`0x00000401`), an Uncertain one `UncertainDataSubNormal | Calculated` (`0x40A40401`) **with its value** (#685), and an empty / null bucket `BadNoData | Calculated` (#681), stamped with its interval start (see below). **Native `TimeAverage` / `Total` rows (#685 — client-visible).** Two mapping defects found by the live AVEVA gate (2026-10-12) are fixed: - **One result per interval in `[Start, End)`.** AVEVA always answers one extra row for the interval *before* the window (`[start − interval, start)` — a real value when data exists there, a no-data row otherwise). It used to be passed through, so a client got N + 1 values, the first stamped before its StartTime. Rows whose interval start lies outside the requested span are now dropped (a row with no start time is placed by its end time, `(lo, hi]`); the span is taken in either order, so the filter does not depend on the read direction. The stream is still drained in full and filtered afterwards — never broken out of. - **An Uncertain interval keeps its value.** AVEVA marks an interval that holds a NULL/Bad sample `opc_quality = 64` (Uncertain) and still supplies its value. It used to be mapped `BadNoData` with the value discarded (`opc_quality >= 192` was the only "has data" test); every Uncertain-family quality (64–191, low byte) now returns the value as `UncertainDataSubNormal | Calculated` — the sub-code the SQL backends give an interval that includes non-good data. Only a Bad-family quality (< 64) is `BadNoData`, null. - ⚠️ **AVEVA's semantics differ from the SQL backends' across a NULL gap.** AVEVA's `TimeWeightedAverage` holds the previous value up to the NULL sample and **excludes the gap time from both numerator and denominator** (live: an interval with 10 good seconds of (5 → 1) answers 3, `Integral` 30); the `SqlServer`/`QuestDb` evaluator bridges across the outage (Part 13 interpolated bounding — 6.08 Uncertain for the same samples, see [Time-weighted aggregates](#time-weighted-aggregates-part-13-671)). Extrapolation past the last sample is `Good` on AVEVA and `Uncertain` on the SQL backends. On all-Good spans the magnitudes match exactly (both sloped/trapezoidal). - ⚠️ **End-to-end correctness also needs the gateway fix historiangw#27:** the gateway sends the aggregate interval to AVEVA in ticks where AVEVA reads milliseconds (10,000× too large), so today a native read returns one row for the whole window (plus the pre-window row, now dropped). OtOpcUa does not work around it; per-interval values return once the gateway is fixed. - **Not verified live:** what AVEVA answers for an interval holding **only** NULL samples (no good time at all). If it marks such a row Uncertain (`opc_quality` 64–191) with a placeholder value, that value is now passed through as Uncertain; the gate only observed Uncertain rows that also held good time. Re-check on the VPN once historiangw#27 is fixed. **Integral unit (#677).** AVEVA's integral is `∫ value dt` (seconds) ÷ the tag's `IntegralDivisor`. The proto3 `double` default for `HistorianTagDefinition.IntegralDivisor` is `0`, so `GatewayTagProvisioner` now sends **`IntegralDivisor = 1`** explicitly on every `EnsureTags` — the value·seconds unit the SQL backends' `Total` uses. A tag provisioned earlier keeps its divisor until its next `EnsureTags` (an upsert). The live magnitude check against AVEVA (a constant 1 integrated over 1 s reads 1) passed on 2026-10-12 at the AVEVA level; end to end it waits on historiangw#27 (above). ⚠️ **Native reads on the SQL backends' interval grid (#691.2, client-visible).** Both directions now use `BucketPlan`, served by **forward** native requests only (`NativeAggregatePlan` — AVEVA anchors a native read's interval grid at the request start, and the gateway refuses `start > end` and `interval <= 0`): - **Reverse** (`StartTime > EndTime`) used to reach the gateway as `start > end` and fail the node (`InvalidArgument`, counted as a backend failure). It now follows #677 like the SQL backends and the raw-evaluated per-sample aggregates (#681): window `(EndTime, StartTime]`, intervals stepping back from `StartTime`, the clipped interval the **earliest**, each stamped with its **later** edge, newest first — one read `[O + I·f, StartTime)` with the interval `I` for the whole intervals (`O = StartTime − n·I`, `f` = 1 when clipped) and a second read over exactly the clipped span `[EndTime, O + I)` with its own width as the interval. A time-weighted value over `[a, b)` and over `(a, b]` is the same integral (two instants apart), so each forward row is the reverse interval's value. - **Forward** windows that are not a whole number of intervals used to send one request whose last AVEVA interval ran **past EndTime**; now the whole intervals `[StartTime, StartTime + (n−1)·I)` are one read and the clipped **latest** interval `[StartTime + (n−1)·I, EndTime)` is its own read over exactly that span. An interval wider than the window is sent as the window, and `interval = 0` as one interval over the window (it used to reach the gateway as 0 and be refused). An empty window (`StartTime == EndTime`) sends nothing. - Stamps match SqlServer/QuestDB (forward: earlier edge, ascending; reverse: later edge, descending), and the clipped interval carries the **Partial** bit unless Bad (Part 13 §5.3.3.2 Case 3, as the SQL backends' time-weighted aggregates — which also mark the intervals the tag's data starts/ends inside; AVEVA's rows carry no data range, so that is not reproduced). Rows are otherwise mapped as before (#685). An interval AVEVA answers no row for is absent (the SQL backends answer it `BadNoData`). - ⚠️ **Interval cap (client-visible):** forward native reads now go through `BucketPlan` too, so a window of more than 100,000 intervals (e.g. 1 s over 2 days) is refused up front with `BadHistoryOperationUnsupported`, as on the SQL backends — it used to be served up to the gateway's own `MaxAggregateIntervals` (1,000,000). Not counted as a backend failure. A clipped interval shorter than 1 ms (a sub-ms window end) is sent as a sub-ms interval request; re-check it in the post-historiangw#27 live leg. - ⚠️ **Until historiangw#27 is fixed, native values are wrong in BOTH directions with a Good status:** AVEVA answers one row per request (the whole span's value), which lands on that request's first interval as a Good, Calculated value while the other intervals are absent. Pinned against a native-mode fake only — re-run the forward and reverse native legs live once the gateway is fixed. Since #681 every native bucket is stamped with its interval **start** (Part 13; it was the interval end) and carries the Calculated info bits (`Good` ⇒ `0x00000401`, no data ⇒ `BadNoData | 0x401`). The live AVEVA gate (2026-10-12) confirmed `start_time` is the interval start (and the `value·s` unit). Previously noted as pending: it reads the aggregate row's `start_time`, which histsdk decodes from a row field (`tail + 43`) its own reverse-engineering notes still mark unverified (HistorianGateway `docs/plans/2026-07-13-archreview2-D-histsdk-residue-design.md`); if that field proves wrong on live AVEVA the stamp is wrong with it. ### Aggregate set and `BadAggregateNotSupported` (#285) > ⚠️ **#681 (client-visible): on the `Gateway` backend the per-sample aggregates are computed from RAW reads.** `Average`, `Count` > (both `BadAggregateNotSupported` since #679 — AVEVA has no arithmetic mean and no sample count), `Minimum` and `Maximum` (were > AVEVA's `MinimumWithTime` / `MaximumWithTime`), and the newly served `Range`, `Start`, `End`, `Delta`, `MinimumActualTime`, > `MaximumActualTime` stream the window's raw samples through the gateway and are evaluated by the SQL backends' shared > `Sql.Common` evaluator (`BucketPlan` + `AggregateEvaluator`), so Gateway answers exactly what `SqlServer`/`QuestDb` answer for the > same samples — values, status (Calculated / Uncertain / Partial bits, #680 Count/Start/End/Delta semantics), interval-START > timestamps, and reverse reads anchored at their later time (#677). What a client sees change: `Minimum`/`Maximum` values and > statuses may differ from AVEVA's own modes; every per-sample result now carries the Calculated info bits; a per-sample read is > bounded by **`ServerHistorian:MaxTimeWeightedRawRows`** (in-window raw rows; past it ⇒ `BadTooManyOperations` for that node — read > a long window over a dense tag in shorter windows). `TimeAverage` and `Total` stay AVEVA-native (`TimeWeightedAverage` / > `Integral`), now stamped with the interval start (was its end) and carrying the Calculated bit; `TimeAverage2`, `Total2`, > `Interpolative`, `PercentGood` and `PercentBad` remain `BadAggregateNotSupported` on Gateway. The served set is the single > static `GatewayHistorianDataSource.ServedAggregates`. **How the Gateway raw evaluation works (#681).** Forward `ReadRaw` reads only (the gateway streams oldest-first and refuses `end < start`; `BucketPlan` assembles a reverse result newest-first), O(buckets) memory, and **every gateway stream is consumed to completion** — breaking out of one cancels the call, and the gateway currently classifies that cancellation as a connection fault and evicts its shared historian session (see the HistorianGateway issue text in the #681 record). - **Window read** `[start − 1 ms, end]`. An AVEVA `Full` raw read emits an *initial-value* row stamped exactly at the query start (histsdk evidence: `Quality = 133`, `OpcQuality = 192`) — the value in effect there, not a stored sample. Starting 1 ms early puts that row outside every bucket. A pre-window row only proves "the tag has earlier data" (start-of-data Partial, Count's in-range zero), and only when its **OPC quality is not Bad** — a "no earlier data" NULL row still carries a value on the wire (histsdk sets `NumericValue = 0` on every numeric row and `StringValue = ""` on a string one) but with a Bad OPC quality (captured as 1, no-data; histsdk string-read-layout.md). - **Earlier-data probe.** When the data range matters (every per-sample aggregate but `Average`) and no non-Bad pre-window row arrived, a 2-row read `[1970-01-01, start)` asks whether the tag has data before the window. Normally this fires only for a tag whose history really begins inside the window. - **Look-ahead.** The end-of-data rules need to know whether data exists after the window: a sample exactly at the window end (forward) comes from the window read; otherwise a 2-row read `[end, now + 1 day]`. In both small reads a row stamped exactly at the query start (a boundary row) or the query end (a possible terminal row) is not a sample, whatever its value. - **Bound.** In-window rows are capped by `MaxTimeWeightedRawRows`; the gateway is asked for `cap + 18` rows, **clamped at the gateway's default ceiling of 1,000,000 when the cap itself is at or under it** (#688 — the default `1,000,000 + 18` used to exceed that ceiling, so a gateway with `AllowStreamingBeyondCeiling=false` refused **every** per-sample read, even a 9-row window, until the cap was lowered; the default config now works against the default gateway). A cap above 1,000,000 is a deliberate choice for a streaming gateway and keeps its `+ 18`. Past the cap the rest of the (bounded) stream is drained, then the node fails `BadTooManyOperations`; a stream that delivers the whole budget cannot be proven complete and fails the same way (so with the default cap a window of more than ~999,982 rows fails, as it would one row later). The gateway's own refusal of the budget (`InvalidArgument` "max_values exceeds the server ceiling of N" — `Historian:ReadLimits:MaxValuesCeiling` lowered below the default, or an unbounded `MaxTimeWeightedRawRows ≤ 0`, with `AllowStreamingBeyondCeiling=false`) is **retried once with N as the budget** (#691.3): a window that fits is served, one that delivers all N rows fails `BadTooManyOperations` as above (not a backend fault). Lowering `MaxTimeWeightedRawRows` to `ceiling − 18` on such a gateway is no longer needed — it only saves the refused round-trip. A raw read's budget is **not** clamped (see [#687](#gateway-raw-reads-return-stored-samples-only-687)). A caller cancellation (client cancel, `HistoryReadDeadline`) is not counted as a backend fault. - **Bucket statistics** are `RawBucketAccumulator`, the C# statement of the SQL backends' per-sample columns — incl. #684: Average / Minimum / Maximum / Range / *ActualTime over **Good** samples only, the Uncertain extremes feeding only the Min/Max status rule, and an Uncertain String/DateTime sample not refusing the Good-value aggregates. A change to those SQL columns must be repeated there. - **Status** is the raw read's quality mapping. C-002: the gateway discards quality on write, so AVEVA history is normally all Good. ⚠️ **Known pre-existing limitation:** `SampleMapper` falls back to AVEVA's own `Quality` field when `OpcQuality` is 0 — but 0 is also the OPC DA Bad value, so a Bad row whose AVEVA `Quality` is ≥ 64 (e.g. 133) maps Uncertain/Good. Raw reads have always done this; since #681 such a row also feeds the per-sample aggregates (as a usable sample). The earlier-data evidence rule reads `OpcQuality` alone and is not affected. **Cost (client-visible).** A per-sample read now moves every raw sample of the window over the gateway instead of one row per interval — notably `Minimum`/`Maximum`, which were native AVEVA modes before #681. Latency and gateway session time grow with the window's sample count, and the cap makes a long window over a dense tag fail: a month of a 1 Hz tag is ~2.6 M samples ⇒ `BadTooManyOperations` at the default 1,000,000 cap (≈ 11.5 days of 1 Hz). Read such windows in shorter slices, or use `TimeAverage` (still native). **Why the time-weighted family is not raw-evaluated.** Part 13 time-weighted aggregates need the real *bounding* samples — the time, value and quality of the latest sample before the window (sloped interpolation across the window start, an outage straddling it). The gateway has no "previous stored sample" read: a raw read streams forward from its start, and AVEVA replaces whatever lies before with the initial-value row stamped at the query start, losing the original time. So `TimeAverage`/`Total` keep AVEVA's native modes (exact to AVEVA's own definition, not provably Part 13), and the other five time-weighted members stay unsupported rather than approximated. **Live leg (open — needs the VPN to `wonder-sql-vd03`):** (1) that AVEVA's raw read emits the initial-value row with a non-Bad OPC quality when earlier data exists and a Bad (NULL) one when none does; (2) that the window read emits no terminal row at its query end (the window read relies on there being none; the small reads ignore one); (3) that a 2-row raw read starting at `1970-01-01` and one ending at `now + 1 day` are accepted and cheap; (4) the gateway's ceiling rejection text still matches "max_values exceeds the server ceiling"; (5) `Minimum`/`Maximum` against live data vs. the old AVEVA modes; (6) `TimeAverage`/`Total` magnitude and the interval-start stamp (`tail + 43`, above). `HistoryAggregateType` carries the wider Part 13 set (Interpolative, TimeAverage, Start, End, Delta, Range, Min/MaxActualTime, Start/EndBound, DeltaBounds, DurationGood/Bad, PercentGood/Bad, WorstQuality, NumberOfTransitions, AnnotationCount, StandardDeviationPopulation, VariancePopulation); `AggregateFunctionMap` (`OpcUaServer`) maps each standard aggregate NodeId (`TimeAverage2` and `Total2` were appended for #671). The **`SqlServer`** and **`QuestDb`** backends serve the seventeen listed in the [aggregate table](#aggregates-processed-reads); everything else (e.g. `StartBound`, `DurationGood`, `StandardDeviationPopulation`) is `BadAggregateNotSupported` for that node. The **HistorianGateway** backend serves twelve (#681 — `GatewayHistorianDataSource.ServedAggregates`): the ten per-sample aggregates from raw reads, plus TimeAverage and Total through AVEVA retrieval modes (`AggregateModeMapper`); for any other member it throws `HistoryAggregateNotSupportedException` without calling the gateway (and since #682 the server refuses it up front), which the HistoryRead path turns into `BadAggregateNotSupported` for that node only (not a generic failure, and not counted as a backend fault). A NodeId outside the mapped set (e.g. `StandardDeviationSample`) is still `BadAggregateNotSupported`. ### Advertised aggregates (`AggregateFunctions`, #682) Each backend declares the aggregates it serves as `IHistorianDataSource.SupportedAggregates` — **the single authority** for two things: - **What the server advertises as HISTORY aggregates.** `HistoryServerCapabilities/AggregateFunctions` (`i=11201`) lists exactly that set. Before #682 it listed the SDK's full 37 regardless of backend. - **What HistoryRead-Processed refuses up front.** An aggregate outside the set is `BadAggregateNotSupported` for that node without a backend call (the backend still throws `HistoryAggregateNotSupportedException` for one as defence in depth — derived from the same table, never a second list). A source whose `SupportedAggregates` is `null` (out-of-tree, against the contract) is treated as the interface default — **all** — by both consumers. ⚠️ **`Server/ServerCapabilities/AggregateFunctions` (`i=2997`) is deliberately NOT trimmed** — it keeps the SDK's full 37 in every configuration, including with no historian. Part 5 defines it as every aggregate the *server* supports, and the server still computes all of them for live data: a MonitoredItem `AggregateFilter` is validated and computed by the SDK's `AggregateManager`, not the historian. The SDK models the same split (`DiagnosticsNodeManager.AddAggregateFunction` always adds to i=2997 and to i=11201 only for historical aggregates). The two are separate folder nodes organizing the same aggregate-function objects. A client choosing a history aggregate should read **i=11201**. | `ServerHistorian:Backend` | i=11201 (history) | i=2997 (server) | Declared by | |---|---|---|---| | `Gateway` | 12 — the ten per-sample aggregates (raw-evaluated, #681) + TimeAverage, Total (native AVEVA modes) | all 37 | `GatewayHistorianDataSource.ServedAggregates` (the per-sample families of `AggregateFamilies` ∪ `AggregateModeMapper.NativeAggregates`, derived from its mapping switch) | | `SqlServer`, `QuestDb` | 17 — the [aggregate table](#aggregates-processed-reads) | all 37 | `AggregateFamilies.Served` (Sql.Common; every member with a family) | | historian disabled (Null source) | **none** | all 37 | `NullHistorianDataSource` | ⚠️ **Client-visible: with no historian configured a processed HistoryRead is now `BadAggregateNotSupported`, not `GoodNoData`.** An aggregate is a statistic the server cannot compute without a history, and the empty i=11201 now says so; Raw / AtTime / Events reads on the Null source still return the empty `GoodNoData` result. How it is wired: `OtOpcUaSdkServer` keeps the SDK's `CreateAggregateManager` (all 37 calculator factories) and then trims only the i=11201 ⇄ aggregate `Organizes` references (both directions) to the node manager's current source — the Null default at boot; the Host's `SetHistorianDataSource` (the backend resolved once from `ServerHistorian:Backend`) re-syncs it right after start, and every later assignment re-syncs both ways. ⚠️ **The calculator factories are deliberately NOT removed**: live aggregate subscriptions keep working for every SDK aggregate (pinned over the wire by `AdvertisedAggregateFunctionsWireTests`). The aggregate-function object nodes themselves stay readable. A boot log line names the advertised history set. ⚠️ **Boot window.** `SetHistorianDataSource` runs right after the SDK server has started (`OtOpcUaServerHostedService.StartAsync`), so for the few milliseconds between the endpoint opening and that call i=11201 is empty and a processed HistoryRead is refused `BadAggregateNotSupported` — the Null default's answer. A client that browses i=11201 in that window sees an empty list; re-browse after connect if it matters. Not changed: the `HistoryServerCapabilities` booleans (`AccessHistoryDataCapability`, `InsertDataCapability`, …) are still whatever the SDK sets — #682 touches only the i=11201 folder (follow-up candidate). ### Known limitations - **Processed and AtTime are single-shot** (no continuation points). Unlike Raw, neither `ReadProcessedDetails` nor `ReadAtTimeDetails` carries a client count cap (`NumValuesPerNode`): the Processed bucket count is deterministic (window / interval) and AtTime returns exactly one sample per requested timestamp, so the single-shot backend returns the complete result in one read and there is no "full page ⇒ maybe more" signal to page on. Returning the full result with no continuation point is spec-conformant. - **Bounds are raw only:** `ReturnBounds` (#691, [Bounding values](#bounding-values-returnbounds-691)) never interpolates a bound — it is always a stored sample or `Bad_BoundNotFound`. - **No modified-value history on the `Gateway` backend.** Modified reads (`IsReadModified = true`) are served by `SqlServer` and `QuestDb` ([Modified-value history](#modified-value-history-part-11), #668); the HistorianGateway exposes no modified/edited-history surface, so on `Gateway` (and with no historian) they return `BadHistoryOperationUnsupported`. ### Redundancy and authorization History reads are served from any node — there is no Primary gate. The `HistoryRead` AccessLevel bit set at materialization admits a node to the read dispatch. With `Security:Acl:Enabled=false` (the default) HistoryRead is **ungated** for any admitted session; with it `true`, the scoped `HistoryRead` grant decides each node (see [security.md](security.md#data-plane-authorization)). **HistoryUpdate is always gated** — see [HistoryUpdate](#historyupdate). --- --- ## SQL Server backend Selected with `ServerHistorian:Backend=SqlServer`. Implemented in `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer` (the project does not reference the Gateway project). All five seams share one connection helper, one schema bootstrapper and one tag catalog, so the schema is ensured once per process and tag ids are cached once. ### Schema Created by the embedded idempotent scripts `Schema/001_initial.sql` and `Schema/002_sample_modified.sql` (mirrored at [`docs/historian/sqlserver/001_initial.sql`](historian/sqlserver/001_initial.sql) and [`docs/historian/sqlserver/002_sample_modified.sql`](historian/sqlserver/002_sample_modified.sql)); `$(Schema)` is the configured schema. The bootstrapper applies every script above the recorded version, in order, under the schema applock. | Object | Purpose | |---|---| | `SchemaVersion(Version, AppliedUtc)` | Applied-script ledger. Required version: **1**; version **2** adds the modification trail (#668). A v1 database is served with the trail features off (#674 — see below). | | `Tag(TagId int identity, Name nvarchar(400) unique, DataType tinyint, EngineeringUnit, Description, CreatedUtc, UpdatedUtc)` | One row per historian tagname. `DataType` is a `DriverDataType`. | | `Sample(TagId, SourceTimeUtc datetime2(7), ValueKind tinyint, NumValue float, IntValue bigint, StrValue nvarchar(max), StatusCode int, ServerTimeUtc)` | **PK `(TagId, SourceTimeUtc)`** — one value per timestamp (Part 11), which makes redelivery idempotent and rules out tie clusters. `StatusCode` is the uint status stored bit-for-bit. | | `Event(EventId nvarchar(64) PK nonclustered, SourceName, AlarmId, AlarmName, AlarmTypeName, Severity, EventKind, Message, UserName, Comment, EventTimeUtc, ReceivedTimeUtc)` | Alarm history. Clustered on `(EventTimeUtc, EventId)`; extra index `(SourceName, EventTimeUtc)`. | | `SampleModified(TagId, SourceTimeUtc, ModifiedId bigint identity, ModificationTimeUtc, UpdateType tinyint, Principal nvarchar(200), ApiKeyId, ValueKind, NumValue, IntValue, StrValue, StatusCode, ServerTimeUtc)` (v2) | The Part 11 [modification trail](#modified-value-history-part-11): one row per sample a HistoryUpdate changed, carrying the **prior** value columns (all NULL for an insert). Clustered PK `(TagId, SourceTimeUtc, ModifiedId)`. ⚠️ No FOREIGN KEY, CHECK or trigger — it is an `OUTPUT … INTO` target, which forbids them; adding one fails every HistoryUpdate. | | `SampleRow`, `TimeRow` table types | TVPs for batched sample writes/merges, AtTime and delete-at-time. | Column mapping per `ValueKind`: `Double` → `NumValue`; `Int64` → `IntValue` + `NumValue` (as a double, for aggregates); `Boolean` → `IntValue` 0/1 + `NumValue` 0/1; `String` → `StrValue`; `DateTime` → `IntValue` (UTC ticks); `Empty` → all NULL. `ValueKind` codes: Empty 0, Double 1, Int64 2, Boolean 3, String 4, DateTime 5. Timestamps are bound as `datetime2(7)` everywhere (full 100 ns precision — never a `DataTable`, which would round to 3.33 ms). **Applying the scripts by hand** (needed when `AutoCreateSchema=false`): create the database, then run each script in order — `sqlcmd -S -U -P -d -I -v Schema=hist -i docs/historian/sqlserver/001_initial.sql`, then the same with `002_sample_modified.sql` (`-v Schema=` supplies the `$(Schema)` placeholder; verified against the docker-dev SQL container). Re-running is a no-op. ⚠️ **Upgrading an existing v1 historian (#668, #674).** With `AutoCreateSchema=true` (the default) the first historian call after the upgrade applies `002_sample_modified.sql` in place (it only creates `SampleModified`; existing samples are untouched). With `AutoCreateSchema=false` a **v1 database is accepted** (#674 — between #668 and #674 it refused every historian call, continuous-historization drains included): writes, reads, events and retention work, while modified-value reads and HistoryUpdate report `BadHistoryOperationUnsupported` without touching SQL, retention skips `SampleModified`, and one Warning per boot names the missing script. Run `002_sample_modified.sql` by hand as above and **restart the node** to enable them (the version is read once per process). Only a schema below version 1 (`missing version 1 (found 0)`) still fails every call. Modifications made before the upgrade have no trail; the trail starts with the first HistoryUpdate after it. ### Writes - **Live values** (`IHistorianValueWriter`): resolve `TagId` through a cache, creating the `Tag` row on a miss (`MERGE … WITH (HOLDLOCK)`; `DataType` from the first non-Empty value's kind, provisioning corrects it later), then `INSERT … SELECT … WHERE NOT EXISTS` through the `SampleRow` TVP. A null timestamp is stamped now; `ServerTimeUtc = SYSUTCDATETIME()`. It returns `false` (never throws) on a SQL failure so the recorder's retry/quarantine owns it. Semantics: [delivery semantics](#outbox-format-v2-and-rollback). - **Alarm events** (`IAlarmHistorianWriter`): per event, insert-if-absent keyed by a deterministic `EventId` = lowercase-hex SHA-256 over the length-prefixed fields `EquipmentPath, AlarmId, EventKind, TimestampUtc.Ticks, User, Comment, Message`. Outcomes: `Ack` on insert or already-present (including a concurrent writer winning a 2627/2601 race); `RetryPlease` on transient SQL errors (connection, timeout, deadlock 1205 …), bootstrap failures, cancellation and any unclassified error; `PermanentFail` (dead-letter) only for data errors SQL will reject forever (truncation 2628/8152, conversion, null, range, constraint). `SourceName` is the event's `EquipmentPath`; `Severity` is the numeric Part 9 value (Low 200, Medium 500, High 700, Critical 900). - **Tag names** are `nvarchar(400)`, and a longer name is **refused, never truncated** (a truncating parameter would resolve it onto whichever tag owns its first 400 characters): `EnsureTags` fails it individually, the live-value writer returns `false` for that tag (a per-tag failure, so the #538 quarantine eventually abandons it — logged as a Warning), the updater answers `BadInvalidArgument` for every item, and reads treat it as not found (Raw empty, AtTime / Processed `BadNoData`). An event read filtered by a source name over 400 characters returns no events. Keep historian tagnames under 400 characters. - **Schema bootstrap failures are cached for 5 s.** With `AutoCreateSchema=false` and the schema missing (or SQL down), every historian call fails fast with the last bootstrap error for 5 s before SQL is tried again, so a missing schema does not serialise every write, read and alarm event behind one round-trip each; each retry that fails logs a Warning. If `sp_releaseapplock` fails after an auto-create, the connection pool is cleared so the session holding the applock is closed rather than returned to the pool. ### Raw reads `ReadRawAsync` returns every sample as stored — outage markers (`Empty` + Bad status) included. The one exception is defensive: a row with no value whose stored status is **not** Bad (only a build from before the writer forced `Empty` samples Bad could have written one) is served with `BadNoData`, on Raw and AtTime alike. | Request | Window | Order | |---|---|---| | `start < end` | `[start, end)` | ascending | | `end < start` (Part 11 reverse) | `(end, start]` | descending | | `start == end` | the sample exactly at that time | — | `NumValuesPerNode == 0` is unbounded (larger values clamp to `int.MaxValue`). An unknown tag is an empty result (`GoodNoData`); reads **never create** tag rows. The node manager's paging is unchanged; because the PK rules out tie clusters, resuming "from the last timestamp inclusive and trimming it" is exact. ### AtTime reads One statement per call: the distinct requested times go in a `TimeRow` TVP and each picks up its previous-or-equal sample `p` and next sample `n` via two `OUTER APPLY`s. A *usable* sample has severity Good or Uncertain **and** a value (an `Empty` sample is an outage marker and counts as Bad whatever its code). Per requested time `T` (the result is stamped `T`): | Case | Result | |---|---| | `p` at exactly `T` | the stored value and status, unchanged | | no `p` (before the first sample), or unknown tag | `BadNoData`, null | | **`p` is Bad** (`T` lies inside an outage) | **null value carrying `p`'s Bad status code** (info bits stripped) — never better than its source | | `p` usable, **`n` Bad** | **`p`'s value held** (not interpolated into the outage), `UncertainDataSubNormal` + Interpolated | | `Double` / `Int64` with a usable `n` | linear interpolation of `NumValue`; `Int64` is computed **through `double`** and rounded to the nearest integer (ties away from zero), keeping the `Int64` type. Status `Good` + Interpolated when both neighbours are Good, else `UncertainDataSubNormal` + Interpolated | | `Boolean` / `String` / `DateTime`, **or `T` after the last sample** | **stepped**: `p`'s value with `p`'s status + Interpolated | | `T` after the last sample **and after now** | `p`'s value held, `UncertainDataSubNormal` + Interpolated (the value is not known to still hold) | Setting Interpolated (or, in aggregates, Calculated) first clears any historian bit the stored status already carries, so a result never has both. ⚠️ Because `Int64` interpolation goes through `double`, magnitudes beyond 2^53 lose precision. ### Events `[start, end)` by `EventTimeUtc`; through the five-argument read **a reversed window (`end < start`) is swapped and returned ascending**. The direction-aware overload with `reverse: true` (what the node manager calls for a reverse read, #675) reads `(lo, hi]` ordered `(EventTimeUtc, EventId)` **descending** — the clustered index scanned backwards. A null `sourceName` means all sources, otherwise `SourceName = @source`. Forward is ordered `(EventTimeUtc, EventId)` ascending. A bounded read fetches `maxEvents + 1` rows and returns a non-null continuation point when the extra row exists; the node manager turns that flag into an OPC UA continuation point (see [Continuation-point paging (Events)](#continuation-point-paging-events-669)). ### Aggregates (Processed reads) > **Two kinds.** The *per-sample* aggregates (Average, Minimum, Maximum, Count, Range, Start/End/Delta, Min/MaxActualTime) are > simple per-sample definitions — no time-weighting, no interval-boundary values — pushed down to SQL. The *time-weighted* aggregates > (TimeAverage, TimeAverage2, Total, Total2, Interpolative; #671 — and the duration-based PercentGood/PercentBad since #680) follow > OPC UA Part 13 with bounding values — see [Time-weighted aggregates](#time-weighted-aggregates-part-13-671). Any `HistoryAggregateType` not in the tables throws > `HistoryAggregateNotSupportedException` ⇒ `BadAggregateNotSupported` for that node (e.g. `StartBound`, `DurationGood`, > `StandardDeviationPopulation`). > > ⚠️ **Release note / migration (#671): `Total` changed meaning on the `SqlServer` and `QuestDb` backends.** It was the plain `SUM` of > the usable samples in the interval, pushed down to SQL; it is now the Part 13 time-weighted integral, `TimeAverage × interval seconds` > (value·seconds), **evaluated in C# from the raw rows** (not pushed down) and therefore subject to `ServerHistorian:MaxTimeWeightedRawRows` > — a long `Total` over a dense tag that used to succeed may now fail `BadTooManyOperations`; read it in shorter windows. This brings > `Total` closer to the `Gateway` backend, whose `Total` maps to AVEVA's native time-weighted `Integral` retrieval mode — but AVEVA's > integral time unit/divisor has not been verified, so the two backends may differ by a constant factor for the same data. A client > that relied on the old sum gets a different number for the same data (e.g. two samples 10 and 20 ten seconds apart no longer total > 30). There is no switch back; `Count` × `Average` reproduces the old sum. > ⚠️ **Release note (#677): reverse processed reads changed — client-visible.** A reverse read (`end < start`) is now anchored at the > request's start (its *later* time) for **every** aggregate, as OPC UA Part 13 and the SDK's own `AggregateCalculator` do: intervals > step back from `start`, the window is `(end, start]`, the truncated interval is the **earliest** one, and each result is stamped with its > interval's **later** edge. Before #677 only `Interpolative` did this; every other aggregate swapped the window, anchored it at the > earlier time (`[end, start)`, the partial interval at the latest end) and stamped each interval with its earlier edge. So a reverse > 80 s → 0 s read at 20 s used to answer `[60,80) [40,60) [20,40) [0,20)` stamped 60/40/20/0; it now answers > `(60,80] (40,60] (20,40] (0,20]` stamped 80/60/40/20 — a sample exactly on an interval edge now belongs to the interval it *ends*, a > sample at `start` is included and one at `end` is not (the same window a reverse raw read uses). Forward reads are unchanged apart > from the new (forward-only) start-of-data Partial bit below. A client that paired reverse results with forward ones by timestamp must shift by one > interval. > ⚠️ **#680 (client-visible): `Count`, `Start`, `End`, `Delta` and the end-of-data Partial bit now follow Part 13 / the SDK** on the > `SqlServer` and `QuestDb` backends: > - **`Count`** counts **Good** samples only (it used to count Good + Uncertain); its status is value-based — Good when every sample in > the interval is Good, the generic **Bad** (`0x80000401` — generic Bad + our Calculated bit; the SDK returns a bare `0x80000000` — value null) when every one is Bad (it used to be `BadNoData`), else > Uncertain. An **empty interval inside the tag's data range** `[first sample, last sample]` is now **`0`, Good** (it used to be > `BadNoData`); an empty interval wholly before the first or after the last sample stays `BadNoData`. Both directions. > - **`Start` / `End`** return the interval's **raw** first / last sample of **any** quality — its own value (null when Bad), its own > stored status (Raw: no Calculated bit) and **its own timestamp** (they used to return the first/last *usable* value stamped with > the interval start under the per-sample status rule). A reverse read returns the same samples: `Start` is still the earliest. > - **`Delta`** (last usable − first usable) is Good unless the interval's first or last sample is not usable (then Uncertain); a Bad > sample in the middle no longer degrades it, and neither does an Uncertain sample. > - **End-of-data Partial:** the interval holding the tag's **last** sample carries the Partial bit for every per-sample aggregate but > `Average` — forward with the interval's earlier edge included, reverse only when the sample lies strictly inside (the SDK's rules) — > so a "live" tag's latest interval is now flagged. > > A client that read `Count` as "usable samples", keyed `Start`/`End` on the interval start, or treated `Count = BadNoData` as "no > samples" must adjust. (`Average`/`Minimum`/`Maximum` were left on Good + Uncertain samples here; #684, below, moved them to Good only.) > ⚠️ **#684 (client-visible): `Average`, `Minimum`, `Maximum`, `Range` and `Minimum`/`MaximumActualTime` use GOOD samples only** on > the `SqlServer` and `QuestDb` backends (they used every *usable* sample — Good **and** Uncertain). Part 13 defines each over "Good > raw values" (§5.4.3.5, .10–.14), independent of `TreatUncertainAsBad` (only `Minimum2`/`Maximum2`, not served, consult it), and the > SDK's `AverageAggregateCalculator`/`MinMaxAggregateCalculator` skip Uncertain values under the server's own `TreatUncertainAsBad = false`. > - An Uncertain sample is **never part of the value** (an interval `Good 6, Good 8, Uncertain 4` now averages 7, was 6; its Minimum is > 6, was 4), and an interval holding **only** Uncertain (and Bad) samples is now **`BadNoData`** (it used to return their value, > Uncertain). > - **Status:** `Average` is value-based (Good only when every sample is Good, else `UncertainDataSubNormal` — unchanged, the SDK's > rule). `Minimum`/`Maximum`/`Range`/`*ActualTime` follow their Part 13 tables: `UncertainDataSubNormal` when a **Bad** sample > exists, or an **Uncertain value lies beyond the reported Good extreme** (below the minimum for Minimum/MinimumActualTime, above the > maximum for Maximum/MaximumActualTime, either for Range; strictly — equal does not count); otherwise **Good**. So an Uncertain > sample inside the Good range no longer makes Minimum/Maximum Uncertain (it used to). > - The SDK applies only the Bad half of that rule (its `ComputeMinMax` collects the Uncertain extremes and never reads them) and answers > Good for an Uncertain value beyond the extreme; ours follows Part 13 there — the one remaining per-sample Uncertain difference, pinned > by `SdkAggregateParityTests`. > - **Non-numeric samples:** these aggregates now answer `BadAggregateInvalidInputs` only for a **Good** `String`/`DateTime` sample > in the interval. An **Uncertain** one is skipped like any Uncertain value — the interval is served from its Good numeric values > with `UncertainDataSubNormal` (the SDK's Average skips it; its Min/Max flags the value it cannot cast). `Delta` reads usable > samples and still refuses any usable non-numeric one. > - `Count` (Good only, #680), `Start`/`End`/`Delta` (usable / any-quality picks) and the time-weighted family (Uncertain time counts as > Good time under `TreatUncertainAsBad = false`, as Part 13 §5.4.3.2 says) are unchanged. The advertised `TreatUncertainAsBad = false` > stays correct: it governs the time-weighted and status calculations; the per-sample aggregates above are defined over Good values > whatever its setting. **Bucketing.** Forward: one result per interval of `[start, end)`; bucket *i* = `[start + i·interval, start + (i+1)·interval)`, start-inclusive / end-exclusive to the tick, the last bucket truncated at `end`, each stamped with its start. Reverse (`end < start`, #677): the window is `(end, start]`; result *k* = `(start − (k+1)·interval, start − k·interval]`, clipped at `end` (the **earliest** interval is the truncated one), stamped with its **later** edge `start − k·interval`, returned latest first. `interval <= 0` ⇒ a single interval `[start, end)`; `start == end` ⇒ no intervals. The bucket index is computed in **100 ns ticks** (not milliseconds): `DATEDIFF_BIG(microsecond, @start, t) × 10` plus a sub-microsecond correction from the two tick digits, divided by the interval in ticks. (`DATEDIFF` counts unit *boundaries crossed*, so a millisecond formulation misplaces samples near a bucket edge, and a nanosecond one overflows past ≈292 years.) A reverse read adds one constant to that elapsed-ticks dividend (`BucketPlan.BucketIndexAdjustTicks`; QuestDB folds it into its base) that re-anchors the grid at the later time and makes the buckets later-edge-inclusive — `BucketPlan.IndexOf` is the C# statement of the math both dialects mirror. A window of more than **100 000 intervals** is refused before any I/O and reported `BadHistoryOperationUnsupported` for that node. Every returned status carries the historian *Calculated* info bit, `BadNoData` included. **Partial bit (#677).** On a **forward** read, every aggregate except plain `Average` sets the **Partial** bit (`0x4`) on a non-Bad result for the interval the tag's data **starts inside** — its earliest stored sample (any quality) lies strictly between the interval's edges, so there is no bounding value before it and the data does not cover the whole interval. This follows the SDK's `m_startOfData` rule, which the SDK turns off for `Average` and in effect applies to forward reads only (a reverse read is fed latest-first, so its "start of data" is the latest sample, past the window). A reverse read therefore sets no start-of-data Partial on the interval holding the *earliest* sample — matching the SDK. An earliest sample exactly on an edge, or before the interval, sets nothing. The time-weighted aggregates also keep setting Partial on the truncated interval (below). **End-of-data Partial (#680).** Every aggregate except `Average` (and `Interpolative`) also sets Partial on a non-Bad result for the interval the tag's data **ends inside** — nothing stored after it: - **forward:** the latest stored sample (any quality) lies at or after the interval's earlier edge and before its later one (Part 13 §5.3.3.2; the SDK's `m_endOfData` rule — the earlier edge is *included*); - **reverse:** the latest sample lies *strictly* inside the interval (the SDK's reverse read keys its `m_startOfData` rule on the first value it is fed — the latest — so a latest sample exactly on an interval edge sets nothing). The time-weighted family does the same (see [Time-weighted aggregates](#time-weighted-aggregates-part-13-671)). Data continuing past the window sets nothing. Two deliberate limits, kept: the per-sample aggregates **never** set Partial on a merely *truncated* interval (only the time-weighted ones do), and **a Bad result never carries Partial** (the SDK sets it on Bad results too). **Data-range queries.** The start/end-of-data rules and `Count`'s in-range empty interval need the tag's **data range**: its earliest stored sample (any quality) at or before the window's later edge and its latest at or after the earlier edge. On **SQL Server** these are one extra result set of two `MIN`/`MAX` subqueries, each a single seek on the `(TagId, SourceTimeUtc)` key. On **QuestDB** they are two `LIMIT 1` statements that **seek the `tag` symbol index** (#683) partition by partition. Before #683 `tag` was unindexed and each was a partition scan whose cost grew with the table, not the window — worst for a stopped tag read over an old window (`LatestSample` scans every later partition) and a late-starting tag (`EarliestSample` scans from the table start). Measured on 53.6 M rows / 120 tags / 61 daily partitions (QuestDB 10.0.1, HTTP round-trip included, median of 7): | Data-range lookup | unindexed warm / cold | indexed warm / cold | |---|---|---| | live tag (either lookup) | 1.5–3 ms / 2–14 ms | 1.1–1.4 ms / 2.5–3.5 ms | | stopped tag, window inside its data (`LatestSample`) | 11 ms / 11 ms | 1.5 ms / 3.5 ms | | stopped tag, window after it stopped (`LatestSample`, no row) | 7 ms / 9 ms | 1.0 ms / 1.9 ms | | late-starting tag (`EarliestSample` from table start) | 7–7.5 ms / 7–10 ms | 1.2 ms / 2–3 ms | With the index an absent side still visits every partition on that side, but as one index-key lookup each rather than a column scan, so it no longer grows with the rows per partition (the tag count); it still grows with the partition count (bounded by the TTL). Alternatives measured and rejected: bounded existence probes ("any sample before the window start / after its end", `LIMIT 1` from the window edge) are cheap only when data *exists* on that side — an absent side is still a full scan, the very case the issue is about; `LATEST ON ts PARTITION BY tag` was the slowest unindexed form (19–32 ms) and no faster than the plain seek indexed; and a per-tag first/last-write side table cannot be kept correct INSERT-only — a TTL partition drop or a HistoryUpdate delete moves the first/last sample with no write to hook. They are read **before** the bucket statement (a separate table reader), so a sample appended in between reaches the buckets but not the range — at worst a stale end-of-data Partial or a `BadNoData` where a `Count` of 0 belongs, never a Good `0` for an interval that has samples (the result of the opposite order). SQL Server reads its range result set first for the same reason. They run only when the answer can change a result (`AggregateEvaluator.NeedsDataRange`): a per-sample read of any aggregate other than `Average`, in either direction (`Count` included); a per-sample `Average` read skips them. **Sample classes.** *Usable* = severity Good or Uncertain **and** a stored value (not `Empty`). *Good* = usable with severity Good. *Bad* = not usable. | Aggregate | Computed over | Value timestamp | |---|---|---| | `Average`, `Minimum`, `Maximum` | `AVG` / `MIN` / `MAX` of `NumValue` over **Good** samples (#684) | bucket start (reverse read: its later edge, as every row marked "bucket start" here) | | `Count` | number of **Good** samples (`Int32`; #680) | bucket start | | `Range` | `Maximum − Minimum` (Good samples) | bucket start | | `Start`, `End` | the **raw** first / last sample of any quality (its type, its own status; value null when Bad — #680) | **the sample's own time** | | `Delta` | last usable − first usable value (numeric samples) | bucket start | | `MinimumActualTime`, `MaximumActualTime` | the **Good** sample (#684) with the smallest / largest `NumValue`; ties ⇒ earliest | **the sample's own time** | Status and edge rules: - An **empty interval** (no samples) ⇒ `BadNoData` — except **`Count`**, which is `0`, Good, when the interval lies inside the tag's data range `[first sample, last sample]` (both edges inclusive, either direction; #680). **Every value aggregate over an interval whose samples are all Bad ⇒ `BadNoData`**; `Count` there is the generic **Bad** (value null). - An interval with samples but **no Good numeric value** (only Uncertain / Bad) ⇒ `BadNoData` for `Average`/`Minimum`/`Maximum`/`Range`/`*ActualTime` (#684). - `Average` is `Good` when every sample in the bucket is Good, else `UncertainDataSubNormal` (value-based, the SDK's rule). `Minimum`/`Maximum`/`Range`/`*ActualTime` are `UncertainDataSubNormal` when the bucket holds a Bad sample or an Uncertain value beyond the reported Good extreme (Part 13 Tables 57–62; strictly beyond), else `Good` (#684). `Count`'s status is value-based (all Good ⇒ Good, all Bad ⇒ Bad, else Uncertain); `Start`/`End` carry the raw sample's own status; `Delta` is Good unless the interval's first or last sample is not usable (then Uncertain) — #680. - `Count`, `Start`, `End` accept any value kind. `Average`/`Minimum`/`Maximum`/`Range`/`*ActualTime` on a bucket that holds a **Good** `String`/`DateTime` sample ⇒ `BadAggregateInvalidInputs` for that interval; an Uncertain one is skipped and makes `Minimum`/`Maximum`/`Range`/`*ActualTime` `UncertainDataSubNormal` (#684). `Delta` is refused by any usable non-numeric sample. - `PercentGood` / `PercentBad` are **duration-based since #680** — see [Time-weighted aggregates](#time-weighted-aggregates-part-13-671). - Everything except `Count` and `Start`/`End` is a `double`; `Int64` samples aggregate as doubles and `Boolean` as 0/1. ### Time-weighted aggregates (Part 13, #671) `TimeAverage`, `TimeAverage2`, `Total`, `Total2`, `Interpolative` and (since #680) `PercentGood` / `PercentBad`, evaluated by the shared `Driver.Historian.Sql.Common/TimeWeightedAggregator.cs` for **both** SQL backends (so `SqlServer` and `QuestDb` agree exactly). **This whole family — `Total` and the percentages included — is evaluated in C#, not pushed down to SQL** (see *Pushdown trade-off* and the raw-row cap below). Bucketing (edges, truncated interval, reverse anchoring, the 100 000-interval cap) is the same as above. > ⚠️ **#680 (client-visible): time-weighted results now follow Part 13 more closely** (`SqlServer` / `QuestDb`; the SDK's own > calculators agree except where noted under *SDK parity*): > - **`TimeAverage` / `Total` status** is `UncertainDataSubNormal` whenever the interval uses an Uncertain value, a segment bridged > across a Bad value, the stepped extrapolation past the last sample, or does not cover the interval — no longer the time-based > rule. **The latest interval of a live tag now reads Uncertain, not Good**, and so does a fully-covered interval bridged across an > outage. > - **`TimeAverage` / `Total` over an interval wholly inside an outage** (data on both sides) now return the bridged value, > Uncertain — they used to return generic `Bad`, value null. > - **`Total` over an interval the data starts inside** integrates only the covered part (it used to be `TimeAverage × interval`). > - **`TimeAverage2` / `Total2` no longer extrapolate past the end of the data** (the last sample, when nothing at or after the window > end exists and it is usable): the interval holding it covers only up to it and is **Partial**; later intervals are **`BadNoData`** > (they used to hold the last value, Good). A one-interval read over the whole history therefore averages the stored data only. > - **`PercentGood` / `PercentBad` are duration-based** (they counted samples), moved to this family — so they are now subject to > `ServerHistorian:MaxTimeWeightedRawRows` (`BadTooManyOperations` past the cap) — and an interval with no sample of its own between > two samples now has a value instead of `BadNoData`. > - **Partial:** set on the forward end-of-data interval (`TimeAverage2`/`Total2`/percentages) and, on a reverse read, on the interval > holding the latest sample when nothing follows it (all of this family but `Interpolative`). > - ⚠️ **Report-by-exception tags (a slowly changing or deadband-filtered value) are the visible case.** A tag whose last stored > sample lies *before* the window and that has nothing after it has, by Part 13, **ended its data**: every interval of > `TimeAverage2` / `Total2` / `PercentGood` / `PercentBad` is **`BadNoData`**, and every interval of `TimeAverage` / `Total` is the > held value, **Uncertain**. When the next sample arrives, a re-read of the same window returns values / `Good` for those intervals > (the held value is then interpolated between real samples, not extrapolated). So a dashboard, a ScadaBridge consumer or any > client that caches processed results for "closed" intervals can see an interval go `BadNoData` → value, or Uncertain → Good, on a > later read — re-read recent intervals, or use `TimeAverage`/`Total` (which always carry a value) for a report-by-exception tag. The > SDK's own calculators behave the same way. Likewise the interval holding a tag's latest sample is Partial and, until a later > sample arrives, `TimeAverage2`/`Total2` there cover only up to it — the comms-loss recovery shape (a single Good sample after an > outage marker, nothing after) answers that sample's value (`Total2` 0), Uncertain + Partial, never Bad/null. **Server defaults, fixed:** `TreatUncertainAsBad = false`, `PercentDataBad = PercentDataGood = 100`, `UseSlopedExtrapolation = false` (`ServerAggregateConfiguration` in the OPC UA server project — the SDK's own `AggregateManager` defaults, which a test pins). **A client's `AggregateConfiguration` (#677):** `UseServerCapabilitiesDefaults = true`, or the four values spelled out equal to these defaults ⇒ served; anything else ⇒ **`BadAggregateConfigurationRejected`** for each node of the request (the source is never called). It used to be silently ignored, so a client asking for, say, `TreatUncertainAsBad = true` got answers computed without it. The check is server-wide (it applies on the `Gateway` backend too). A client sending an all-zero structure without `UseServerCapabilitiesDefaults` is asking for `PercentDataBad = PercentDataGood = 0` and is refused; the SDK's default-constructed `AggregateConfiguration` sets `UseServerCapabilitiesDefaults = true` and is served. **Bounding values.** Besides the window `[start, end)`, each read fetches four single-row seeks: the latest *usable* sample before `start`, the latest sample of any quality before `start` (they differ when an outage straddles the start), the first sample of any quality at or after `end`, and the first *usable* one at or after `end`. So a bucket with no samples of its own, between two samples, still gets a value. **The signal.** Between a usable sample `p` and the next sample `s`: both `Double`/`Int64` ⇒ **sloped** (linear), segment quality the worse of the two; either `Boolean` ⇒ **stepped** (`p` held), quality `p`'s; `s` Bad ⇒ `p` held up to the outage (never interpolated into it — the AtTime rule), Uncertain. From a Bad sample (incl. an `Empty` outage marker) to the next sample the signal is **Bad** (outage) — an outage marker with nothing after it runs to the window end; before the first sample there is **no data**; after a *usable* last sample with nothing at or after the window end the data has **ended** (#680, Part 13 "After End of Data": no extrapolation). That is the *simple-bounding* signal. The *interpolated-bounding* signal skips Bad samples: it joins consecutive usable samples (sloped / stepped) straight across an outage, and holds the last usable value to the window end (stepped extrapolation). | Aggregate | Value | Bad intervals | |---|---|---| | `TimeAverage` | ∫ interpolated-bounding signal ÷ its defined duration | interpolated across (Part 13) | | `Total` | ∫ interpolated-bounding signal over its defined part (value·s) — `TimeAverage × interval seconds` on a covered interval; only the covered part when the data starts inside (#680) | interpolated across | | `TimeAverage2` | ∫ simple-bounding signal over non-Bad time ÷ non-Bad duration | excluded | | `Total2` | ∫ simple-bounding signal over non-Bad time (value·s) | excluded | | `Interpolative` | interpolated-bounding value **at the interval start** (reverse read: see below) | interpolated across | | `PercentGood` / `PercentBad` | `100 ×` Good / Bad time ÷ interval duration (#680, Part 13 §5.4.3.31-34) — see below | counted as Bad time | **Status, `TimeAverage` / `Total`** (interpolated bounding, #680 — Part 13 "Uncertain if Bad values are skipped or Uncertain values are used", and the SDK's `AverageAggregateCalculator`): - No defined value anywhere in the interval (before the first usable sample) ⇒ `BadNoData`. - Any part of the interval uses an Uncertain value (either end of a sloped segment, the start of a stepped one), lies on a segment bridged across a Bad sample, is the stepped extrapolation past the last usable sample, or is not covered (the data starts inside it) ⇒ `UncertainDataSubNormal`. A segment bridged across an outage marks **every** interval it touches, not just the one holding the Bad sample. - Otherwise ⇒ `Good`. **Status, `TimeAverage2` / `Total2`** — the SDK's `AggregateCalculator.GetTimeBasedStatusCode` (inspected in `Opc.Ua.Server` 1.5.378.106) under the fixed defaults. With `TreatUncertainAsBad = false`, **Uncertain time counts as Good time** (Uncertain samples, a value held up to an outage). The duration is the interval's, cut at the end of the data: - After the end of the data ⇒ `BadNoData` (Part 13 "After End of Data"); the interval the data ends inside covers only up to the last sample and is **Partial** (forward). When it has no non-Bad time before the last sample — the data ends exactly on its start, or the last sample is the first usable one after an outage (comms-loss recovery), or the tag's only sample — there is nothing to cover and the value is the last sample's (`Total2` 0): `Good` only when the data ends exactly on the interval start with a Good sample, otherwise `UncertainDataSubNormal` (the SDK answers the same, modulo its 1 ms artifact). - Every instant of the (covered) interval non-Bad ⇒ `Good`. - Some non-Bad time but not all (an outage, or time before the first sample) ⇒ `UncertainDataSubNormal`. - No non-Bad time but some outage time (an interval wholly inside an outage) ⇒ generic **`Bad`** (`0x80000000`), value null — Part 13's PercentDataBad rule. (The SDK answers `BadNoData` there; see *SDK parity*.) - No data at all ⇒ `BadNoData`. For all four, a usable `String`/`DateTime` sample feeding the interval ⇒ `BadAggregateInvalidInputs`. **`PercentGood` / `PercentBad`** (#680, Part 13 §5.4.3.31-34 — they used to count samples). The interval is cut into **stepped regions**, each with the status of the sample that starts it: Good regions count toward `PercentGood`, Bad ones (incl. outage markers) toward `PercentBad`, Uncertain ones toward neither (so the two sum to < 100 when Uncertain data is present). The region that starts an interval before its first own sample takes the status of the *simple bounding value* there (the SDK's `GetSimpleBound`): Bad after a Bad sample, Uncertain on a segment that runs into an outage or slopes to/from an Uncertain sample, else the earlier sample's. Time before the tag's first sample is **Bad**. When the data ends (a usable last sample, nothing at or after the window end) its region runs to the **end of its own interval** (Part 13 — the SDK gives it 1 ms) and later intervals are `BadNoData`; an outage marker with nothing after it is Bad to the window end. An interval that ends before the first sample ⇒ `BadNoData`. Any value kind is accepted; the status is `Good` (plus Calculated / Partial). Value = `100 × region time ÷ interval duration`. All carry the Calculated bit. A non-Bad result also carries the **Partial** bit (`0x4`, `AggregateBits.Partial`) — never on a Bad result — for: the **truncated interval** (shorter than the processing interval — the latest one forward, the earliest one reverse; Part 13 §5.3.3.2 Case 3 requires it — the SDK does not set it, a deliberate difference); the interval the tag's **data starts inside** (#677, forward, see *Partial bit* above); on a **reverse** read, the interval the **latest** sample lies strictly inside when nothing follows it (where a reverse read's data starts — the SDK's `m_startOfData`, keyed on the first value it is fed, #680); and, forward, the **end-of-data** interval of `TimeAverage2` / `Total2` / the percentages (the SDK's `m_endOfData` rule, which it skips for an extrapolated `TimeAverage` / `Total`). Timestamp = interval start (reverse: its later edge); value `double` (Boolean 0/1). > **A live tag's latest interval (#680 — changed, client-visible).** Past the tag's last stored sample `TimeAverage` / `Total` hold the > last value to the interval end (stepped extrapolation), and that interval now reads **`UncertainDataSubNormal`** — the extrapolated > part is not data. (It used to read `Good`, because the time-based rule counted the extrapolated time as Good time.) `TimeAverage2` / > `Total2` do not extrapolate: they average / integrate only up to the last sample and set **Partial**; any later interval is > `BadNoData`. A `Good` latest interval therefore now does mean the data covered it — for these aggregates, the latest interval of a > tag that has not yet received a sample at or after the request's end time is never `Good` (`TimeAverage`/`Total`) or always > `Partial` (`TimeAverage2`/`Total2`, forward). If it was truncated by the request's end time it also carries **Partial**. **Status, `Interpolative`** (point-based, not the PercentDataGood rule): - A usable sample exactly at the point ⇒ that value with **its own stored status** (no historian bit). - No usable sample at or before the point ⇒ plain `BadNoData`. - Otherwise the value with the **Interpolated** bit: - `Good` on a segment whose defining samples are Good; - `UncertainDataSubNormal` inside or up to an outage (bridged) and past the last sample (extrapolated). **Reverse reads (`end < start`).** Part 13 anchors a reverse read at the request's start, which is the later time — for every aggregate since #677. `Interpolative`'s points are `start, start − interval, …`, latest first: a reverse read 10:00 → 09:00 at 1 h returns the value at **10:00**, not 09:00, and no point ever falls before the window. The interval aggregates use the intervals `(start − (k+1)·interval, start − k·interval]`, the earliest one truncated, each stamped with its later edge. **SDK parity (#677).** `SdkAggregateParityTests` (unit, `Sql.Common.Tests`) and `ProcessedConformance.Processed_reads_match_the_SDK_aggregate_calculators` (both SQL backends, through the real SQL) run the pinned SDK's own calculators (`AverageAggregateCalculator`, `MinMaxAggregateCalculator`, `CountAggregateCalculator`, the base `AggregateCalculator` for Interpolative, `StatusAggregateCalculator` for PercentGood/Bad) over the same samples, forward and reverse, and compare every interval's timestamp, value, status code and Partial bit. The per-sample comparison covers Average, Minimum, Maximum, Count, Range, Start, End, Delta and Minimum/MaximumActualTime; the time-weighted one TimeAverage, TimeAverage2, Total, Total2, Interpolative and PercentGood/Bad (#680), including datasets whose data ends inside the window and with an interval wholly inside an outage; reverse reads over data that starts inside the window are compared too. Known differences that remain (tracked as #680) — documented and masked in the tests, deliberately not changed here because forward results were to stay as they are — are: an Uncertain value beyond the Good extreme makes Minimum/Maximum/Range/*ActualTime Uncertain here (Part 13), Good in the SDK (#684 aligned the values — Good samples only — and every other Uncertain case); the SDK sets Partial on Bad per-sample results too, ours never (#680 closed the Start/End, Delta, Count and end-of-data Partial differences). Time-weighted differences (#680 closed the TimeAverage/Total status, `Total` coverage, end-of-data and PercentGood/Bad ones), each deliberate: ours flags a truncated interval Partial (Part 13 §5.3.3.2 Case 3), the SDK does not; ours carries the Calculated bit on a `BadNoData`; `TimeAverage2`/`Total2` over an interval wholly inside an outage are generic `Bad` here (Part 13's PercentDataBad rule), `BadNoData` in the SDK; in the end-of-data interval the SDK gives the region after the last sample 1 ms (so `Total2` there differs by ~1 ms × value and the percentages by a lot — ours follow Part 13: `TimeAverage2`/`Total2` stop at the last sample, the percentages' region runs to the interval end); and on a reverse read the SDK's backward slice files a sample exactly on an interval's later edge as its early bound — so a reverse `Interpolative` at such a point comes back Interpolated (Uncertain if a Bad sample follows) and a reverse `TimeAverage`/`Total` interval ending on it takes the status of the segment beyond it — where ours, like the SDK forward, returns the raw sample and keeps the interval's own segments (an SDK defect: its forward and reverse answers differ). Three more, kept deliberately (#680 review): an **outage marker as the last sample** (nothing after it) runs Bad to the window end here — `TimeAverage2`/`Total2` generic `Bad` and the percentages 100 % Bad after it, no end-of-data Partial on the interval holding it, and `TimeAverage`/`Total` Uncertain over the span the last usable value is held up to the marker (no Good value follows ⇒ extrapolated) — where the SDK treats the data as ended at the marker (`BadNoData` after it, Partial on that interval, the held span Good): the tag is in an outage, not out of data. **Booleans** are stepped here (a non-analog value is held; the simple bound next to an Uncertain sample takes the earlier sample's status), while the SDK interpolates them as numbers — e.g. an interval that is `false` for its first half and `true` for its second averages 0.5 here, 1 in the SDK, and a PercentGood region ahead of an Uncertain Boolean is Good here, Uncertain in the SDK. Each is pinned by `TimeWeightedAggregatorTests` and by a parity dataset (`OutageLast`, `BoolSwitch`) whose masks are documented in `SdkAggregateParityTests`. Two more, found by the #680 live gate (both cosmetic, both kept deliberately): when an interval's **Minimum/Maximum sample sits exactly on the interval's start**, the SDK reports that result as a raw value (no Calculated/InfoType bits, e.g. `0x00000000` / `0x40A40000`) while ours keeps the Calculated bit (`0x00000401` / `0x40A40401`) — value and timestamp are identical, and ours is consistent with every other interval of the same read. And on an **integer-typed tag** (e.g. a `UInt32`) the SDK casts interpolated and sloped bounding values back to the source integer type, so its `Interpolative` returns `758` where ours returns `758.0459…`, and its `TimeAverage`/`TimeAverage2`/`Total`/`Total2` differ from ours by the truncated bound values (~0.002–0.02 on the gate's data). Ours always computes and returns `Double` (as stated above for every aggregate but Count and Start/End), which is the exact Part 13 value; the parity datasets are all `Double`, so this one is not masked there but recorded here. Two more, found by live gate 9 (both kept deliberately): **Minimum/MaximumActualTime carry the Calculated bit on every interval** (`0x…401`, `0x…405` with Partial) where the SDK returns the picked raw sample's own status (`0x…000`, `0x…404`) — the same value and timestamp, and the same treatment as the Min/Max-at-interval-start case above; masked in `SdkAggregateParityTests` (class remarks). And an interval holding a **Good String (or DateTime) sample** is `BadAggregateInvalidInputs` for Average/Minimum/Maximum/Range/ *ActualTime here (the documented #684 refusal rule — a Good non-numeric value is not a number to aggregate), while the SDK skips the string and serves the numeric samples (e.g. Average 4 Good); an Uncertain non-numeric sample refuses nothing on either side. **Pushdown trade-off.** Unlike the per-sample aggregates, the time-weighted family is **not** pushed down to SQL: the backend streams the window's rows plus the four bounds (one batch of five result sets on SQL Server; a bounds `UNION ALL` + a window select on QuestDB) and the evaluator splits segments at bucket edges in C#. Network traffic is therefore O(samples in the window) rather than O(buckets); evaluation memory stays O(buckets) (rows are streamed, never materialised). The alternative — the same Part 13 segment interpolation, edge splitting and two bounding definitions written twice, in T-SQL and QuestDB SQL, and kept bit-identical — was judged the larger risk. For very long, densely sampled windows prefer the pushed-down `Average` (or a coarser read) over `TimeAverage`. **Raw-row cap.** The window query is limited to `ServerHistorian:MaxTimeWeightedRawRows` + 1 rows (default 1 000 000, shared by both SQL backends; `TOP (@maxRows)` / `LIMIT $4`). When that extra row arrives the read stops and the node fails with **`BadTooManyOperations`** — for that node only; it is not counted in the backend's health failures. Without the cap a one-year `Total` on a 1 Hz tag (≈31 M rows) would hold a pooled connection until `HistoryReadDeadline` and then return `BadTimeout`. The four bound rows do not count toward the cap, and the per-sample aggregates are never capped. Since #680 `PercentGood`/`PercentBad` are in this family and **are** capped (they used to be pushed down). Raise the cap, or read in shorter windows. ### Health The read client reports `GetHealthSnapshot()` (query/failure counters, consecutive failures, `LastError` — e.g. a missing schema under `AutoCreateSchema=false`). A SQL error on a read surfaces as `BadHistoryOperationUnsupported` for **that node only**; a cancelled read (deadline, client abort) is not counted as a backend fault. ### Retention (`RetentionDays`) `hist.Sample`, `hist.SampleModified` and `hist.Event` grow without bound unless `ServerHistorian:SqlServer:RetentionDays` is set. QuestDB already has an equivalent TTL (`QuestDb:RetentionDays`); the Gateway backend's retention is the AVEVA Historian's own. - **What runs.** `HistorianRetentionService` (a hosted service, driver-role nodes, Backend=`SqlServer` with `RetentionDays` set; inert otherwise) ticks every `PurgeInterval` (first pass at start-up) and calls the backend's `IHistorianRetention.PurgeAsync`. - **Gate vs. mutual exclusion.** Each tick consults `IHistorizationRoleView.ShouldRecordHistorizedValues` (the capture verdict; a standby skips, an unknown role runs). That verdict is **pair-scoped** while the purge is **database-scoped**, so the gate is only a load reducer: the `sp_getapplock` below is the actual mutual exclusion. Pairs sharing one historian database, and both nodes of a pair on their boot-time pass, simply serialise on it. Set `RetentionDays` **uniformly on every node that shares a historian database** — if they differ, the smallest window wins. - **How it deletes.** Cutoff = `UtcNow - RetentionDays` on source time. Samples are purged **per tag** with `DELETE TOP (@n) ... WHERE TagId = @id AND SourceTimeUtc < @cutoff` (a seek on the clustered `(TagId, SourceTimeUtc)` key; a global time predicate would scan) — and, in the same per-tag pass, the modification trail with `DELETE TOP (@n) FROM SampleModified WHERE TagId = @id AND SourceTimeUtc < @cutoff` (#668; same **source-time** cutoff, clustered seek; the purge itself never writes a trail row) — then events with `DELETE TOP (@n) ... WHERE EventTimeUtc < @cutoff` (seek on `CX_Event_Time`), each looped until a batch comes back short. Cancellation is honoured between batches. The `hist.Tag` catalogue is never purged. - **Shared databases.** Several pairs may point at one database; a session `sp_getapplock` (`OtOpcUa.Historian.Retention.`, timeout 0) lets one purger run and the others skip that tick (Debug log). - **Observability.** Counter `otopcua.historian.retention.purged{table=Sample|SampleModified|Event}`; a failed pass logs a Warning and retries next interval — it never affects capture or reads. - **Schema v1 (#674).** On a v1 schema (no `SampleModified`, `AutoCreateSchema=false`) the trail purge is skipped and omitted from the pass results; samples and events purge normally. - **Not done: partition switching.** For very large installs (billions of rows) a partitioned `hist.Sample` with sliding-window `SWITCH`/drop is cheaper than row deletes; the purger is deliberately simple batched deletes. Another table is added by one more `DeleteLoopAsync` call in `SqlServerHistorianRetention.PurgeAsync` (as `hist.SampleModified` was, #668). ### Docker-dev recipe The docker-dev `sql` container (`otopcua-dev-sql-1`, host port `14330`, `sa` with the committed dev password from `docker-dev/docker-compose.yml`) already hosts the ConfigDb; give the historian its own database on the same instance. ```bash # 1. Create the database (the backend creates the schema + tables on first use, not the database). sqlcmd -S localhost,14330 -U sa -P 'OtOpcUa!Dev123' -C -I -Q "CREATE DATABASE OtOpcUaHistorian" # 2. Point the central nodes at it. Add to the `environment:` of central-1 AND central-2 in docker-dev/docker-compose.yml # (or a compose override), rebuild the images (the rig image goes stale), then `up -d`: # ServerHistorian__Enabled: "true" # ServerHistorian__Backend: "SqlServer" # ServerHistorian__SqlServer__ConnectionString: "Server=sql,1433;Database=OtOpcUaHistorian;User Id=sa;Password=OtOpcUa!Dev123;TrustServerCertificate=True" # ContinuousHistorization__Enabled: "true" # ContinuousHistorization__OutboxPath: "/app/data/historization-outbox" # (ServerHistorian__Endpoint / __ApiKey are not used by this backend.) # 3. Mark a rig tag historized ("isHistorized": true in its TagConfig), deploy, then watch rows land: sqlcmd -S localhost,14330 -U sa -P 'OtOpcUa!Dev123' -C -I -d OtOpcUaHistorian \ -Q "SELECT t.Name, COUNT(*) n, MAX(s.SourceTimeUtc) last FROM hist.Sample s JOIN hist.Tag t ON t.TagId=s.TagId GROUP BY t.Name" ``` Only the Primary of a pair records, so the row count advances once per published value, not twice. To watch retention, add `ServerHistorian__SqlServer__RetentionDays: "1"` and `ServerHistorian__SqlServer__PurgeInterval: "00:00:30"`, insert a `hist.Sample` row with an old `SourceTimeUtc` for an existing tag, and see it vanish within a tick while recent rows stay. Read it back with `Client.CLI historyread` and rewrite it with `Client.CLI historyupdate` ([Client.CLI.md](Client.CLI.md)). The password above is the committed docker-dev credential, not a secret; use `${secret:…}` / environment for anything real. ### Integration tests `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests` (bootstrap incl. two concurrent bootstrappers, every kind round-trip, redelivery, raw / AtTime / aggregate / event reads, every update result code) is gated on the environment variable **`HISTORIAN_SQL_TEST_CONNECTION`** — a connection string with `CREATE`/`DROP DATABASE` rights. Absent ⇒ every test skips. Present ⇒ each run creates a throwaway `OtOpcUaHistorian_T_` database and drops it on dispose. ```bash export HISTORIAN_SQL_TEST_CONNECTION='Server=localhost,14330;User Id=sa;Password=OtOpcUa!Dev123;TrustServerCertificate=True;Encrypt=False' dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests ``` --- ## QuestDB backend Selected with `ServerHistorian:Backend=QuestDb`. Implemented in `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb` (pure logic shared with the SQL Server backend lives in `Driver.Historian.Sql.Common`). All access is the **Postgres wire protocol via `Npgsql`** (no ILP, no extra package). Design: [plans/2026-10-10-questdb-historian-design.md](plans/2026-10-10-questdb-historian-design.md); the spike that pinned every QuestDB-specific form: [plans/2026-10-10-questdb-spike.md](plans/2026-10-10-questdb-spike.md). Pinned image: `questdb/questdb:10.0.1`. QuestDB serves every value kind (`Double`, `Int64`, `Boolean`, `String`, `DateTime`, `Empty`), stores the real status, records comms-loss outage markers, serves the seventeen aggregates (time-weighted ones through the same shared evaluator, so results are identical) and supports **HistoryUpdate** (soft delete), exactly like `SqlServer`. The same conformance suite (`tests/Drivers/Historian.Conformance`) runs against both backends; the differences below are the only ones. ### Configuration (`ServerHistorian:QuestDb`) Read only when `Backend=QuestDb`; bound by `HistorianBackendRegistry.BindQuestDb`, validated (consumer-gated, like `SqlServer`) by `QuestDbHistorianOptions.Validate()`. The gateway keys (`Endpoint`, `ApiKey`, `UseTls`) are not required. | Key | Default | Validation | Meaning | |---|---|---|---| | `QuestDb:ConnectionString` | `""` | **Required** (non-blank) | Npgsql connection string for QuestDB's PG-wire port, e.g. `Host=questdb;Port=8812;Username=admin;Password=quest;Database=qdb;ServerCompatibilityMode=NoTypeLoading`. `ServerCompatibilityMode=NoTypeLoading` skips the `pg_type` load; every query form was verified with it. Supply via env `ServerHistorian__QuestDb__ConnectionString` / `${secret:}` for anything real. | | `QuestDb:TablePrefix` | `hist_` | `^[A-Za-z0-9_]+$` | QuestDB has no schemas, so the four tables are `{prefix}sample`, `{prefix}event`, `{prefix}tag` and `{prefix}sample_modified` (#668). Spliced into DDL/DML, hence the regex. | | `QuestDb:RetentionDays` | unset | `>= 1` when set | Applied as table TTL to `{prefix}sample`, `{prefix}event` and `{prefix}sample_modified` (see [Retention](#retention-ttl)). Unset keeps data forever. | | `QuestDb:CommandTimeout` | `00:00:30` | `> 0` | Per-command timeout, rounded up to whole seconds (minimum 1 s). | | `QuestDb:AutoCreateSchema` | `true` | — | `true`: create the tables (and reconcile the TTL) on first use. `false`: only verify the four tables exist (⚠️ an install from before #668 lacks `{prefix}sample_modified` — create it with the DDL in `QuestDbTables.CreateSampleModified` before upgrading, or every call fails); if one is missing every historian call fails with `HistorianSchemaMissingException` (health `LastError`, no crash) and no TTL is touched. A failed bootstrap is not latched: it is re-thrown for 5 s without touching QuestDB, then retried. | | `QuestDb:VisibilityTimeout` | `00:00:05` | `>= 0` | How long a write waits for its rows to become visible to queries ([WAL visibility](#wal-visibility-and-the-read-your-writes-wait)). `0` disables the wait. | ⚠️ **Connection pool size.** QuestDB's default limit is **64** concurrent PG-wire connections (`pg.net.connection.limit`), while Npgsql's own default `Maximum Pool Size` is **100**. A busy node (HistoryRead batches, the recorder, the updater and alarm writes all share one pool) could therefore be refused by QuestDB before Npgsql's pool is full. **When the connection string does not set `Maximum Pool Size` (or `MaxPoolSize`), the backend now caps it at 24**, so the two nodes of a redundant pair sharing one QuestDB stay under the server limit together (2 × 24 = 48). An explicit value in the connection string is always kept as written — including one above 64, since QuestDB's limit is server-configurable. The effective value is logged once at Information when the backend first builds its pool. ### Tables All are **WAL tables**; the first three carry **`DEDUP UPSERT KEYS`**, so re-sending a row with the same key replaces it (the upsert is what makes HistoryUpdate Replace work; the writer's own insert-if-absent semantics are built on top, below). The modification trail is append-only. | Table | Key | Partitioning | |---|---|---| | `{p}sample` (`ts, tag, kind, num_value, int_value, str_value, status, deleted, server_ts`; **`tag` is a `SYMBOL INDEX`**, #683) | `(ts, tag)` | `PARTITION BY DAY` (TTL when configured) | | `{p}event` (one-for-one with `hist.Event`) | `(ts, event_id)` | `PARTITION BY DAY` (TTL when configured) | | `{p}tag` (`ts, name, data_type, eng_unit, description, updated_utc`) | `(ts, name)` with `ts` fixed at `1970-01-01` — an upsert by name | `PARTITION BY YEAR`, never expires | | `{p}sample_modified` (`ts` = the sample's source time, `tag, modified_id, modification_ts, update_type, principal, api_key_id, kind, num_value, int_value, str_value, status, server_ts`) (#668) | none — append-only, no DEDUP; `modified_id` (process-monotonic ≈ UTC ticks) orders a timestamp's trail | `PARTITION BY DAY` (TTL when configured). A NULL `status` marks "no prior value" (QuestDB `BYTE` has no NULL, so `kind` is 0) | **The `tag` index (#683).** Every read filters on one tag; without a symbol index QuestDB answers `tag = x` by scanning the tag column of every partition in the time range, which made the processed-read [data-range queries](#aggregates-processed-reads) of a stopped or late-starting tag scan every partition on one side of the window. A table created before #683 has no index: with `AutoCreateSchema=true` the bootstrapper adds it on first use (`ALTER TABLE {p}sample ALTER COLUMN tag ADD INDEX`, a WAL transaction; idempotent — re-sending it on an indexed column is harmless, so a pair booting together is safe — and a boot that finds the column indexed sends nothing). Building it is a one-off WAL apply, measured **~3 s for 68 M rows / 63 daily partitions**, during which writes queue and reads keep scanning. With `AutoCreateSchema=false` a missing index only logs a Warning naming that statement — reads stay correct, just slower. With `AutoCreateSchema=true` an ALTER refused up front (e.g. no DDL rights) is likewise only a Warning — the bootstrap still succeeds and the next boot retries; a failure at **WAL-apply** time is not observed by the bootstrapper (QuestDB would suspend the table, visible in `wal_tables()`) — the same blind spot as the TTL ALTER. Cost: ~12 % more disk (0.5 GB on a 4.2 GB 53 M-row table); no measurable change in live-append, O3 back-fill or bulk-load apply time beyond run-to-run noise (a first bulk load into an empty table measured ~40 % slower). Out-of-order writes (back-fill, HistoryUpdate, outbox redelivery) rewrite the index files of every partition they touch along with the column files, so their apply cost carries the index too. On a 120-tag table the per-sample / raw / time-weighted window queries were unchanged or slightly faster (the tag filter becomes an index scan), but **a tag holding a large share of the table can be ~20 % slower on long windows**: the index scan is single-threaded where the unindexed filter is a multi-threaded JIT scan (a review measurement: 4 tags / 30 M rows, 245–294 ms indexed vs 195–218 ms unindexed; with 1000 tags, 9 ms vs 13 ms — the index wins as the tag's share of each partition shrinks). **Deliberately not indexed:** `{p}sample_modified.tag` — read only by `IsReadModified` HistoryRead over a window, an append-only trail a tiny fraction of the sample table's size, with no "nearest row outside the window" lookup to make it scan partitions; and `{p}event.source` — event reads are window-bounded (`ts` range, optionally `source = x`) and never seek beyond the window, and the event table is far smaller than the sample table. Neither has the unbounded-scan shape #683 fixes, so neither pays the index's disk and O3 cost. `status` holds the uint OPC UA status code **unsigned** in a `LONG` (so "Bad" is `status >= 2147483648`, not `< 0`); `int_value` also holds Booleans as 0/1; `kind` is the `HistorianValueKind` in a `BYTE`. Tag names are limited to 400 characters (longer is a per-tag failure, never truncated). `{p}tag` is the **only** "is this tag known" authority (`QuestDbTagCatalog`): reads never create a tag, Insert/Update and the live writer do, Replace and the deletes do not. ### Timestamps: microseconds QuestDB's designated timestamp is a microsecond. Every inbound `DateTime` is normalised by `QuestDbConnection.Round` (Unspecified ⇒ UTC, Local ⇒ UTC, ticks truncated to a multiple of 10, `MinValue`/`MaxValue` nudged inward because Npgsql would send them as ±infinity) and bound only through `BindTimestamp`. Consequences: - Two live samples for one tag **less than 1 µs apart** collapse to one; **the first wins**. A batch is deduplicated client-side after rounding. - **Raw and Processed results carry stored (µs) timestamps.** A sub-µs request bound is rounded, and the rounded start *is* the request start, so a Processed bucket's timestamp is the rounded window start (a sub-µs label divergence from `SqlServer`, which labels with the exact start; values and bucket membership are identical). - **AtTime** evaluates at the rounded time but **stamps each result with the time the client asked for**, so a client can match answers to requests. Stored data is µs-only, so a sub-µs request adjacent to a sample is an exact hit. - `IHistorianDataSource.TimestampResolutionTicks` is **1** by default and **10** on QuestDB. HistoryRead-Raw paging steps off a drained timestamp by that amount (forward `T + resolution`, reverse `T − resolution`). Stepping by 1 tick would be truncated back to `T` and a one-value-per-page forward read would loop forever. Same-timestamp ties (legal on the Gateway backend) are still paged within the timestamp. - **Pre-1970 timestamps cannot be stored** (QuestDB: `designated timestamp before 1970-01-01 is not allowed`; the whole INSERT fails). So: - the **live writer** drops such samples with at most one Warning per tag per 10 minutes (a device with an unset clock would otherwise warn every drain pass; every dropped sample is still counted in `DroppedPre1970`) — a retry could never succeed, and sending them would push the tag into the #538 quarantine; - **HistoryUpdate** Insert / Replace / Update of such an item returns `BadOutOfRange` (nothing is sent); `DeleteAtTime` returns `BadNoEntryExists`; `DeleteRaw` finds nothing; - an alarm event with such a time is a `PermanentFail` (dead-letter); - **read bounds before 1970 work** (a Raw/Processed window may start before the epoch; bounds are never clamped, only `MinValue`/`MaxValue` nudged). - **`Int64` `long.MinValue`.** QuestDB's `LONG` NULL is `long.MinValue`, so a stored `FromInt64(long.MinValue)` has `int_value` NULL. The read path repairs it (`kind = Int64 ∧ int_value IS NULL ∧ num_value = −2^63` ⇒ `long.MinValue`); without the repair it would read back as `Empty`/0. Likewise `NaN`/`±Infinity` doubles read back as NULL, so they are stored as `Empty` + `BadOutOfRange` (live) or rejected `BadInvalidArgument` (HistoryUpdate), as on `SqlServer`. ### Live writes: insert-if-absent, tombstones included The recorder's writer (`QuestDbHistorianValueWriter`) has the same first-wins redelivery contract as SQL Server: 1. group by tag, round to µs, drop in-batch duplicates (first wins), drop pre-1970 samples; 2. read the timestamps that already exist for the tag (one range query per cluster of nearby times, **`deleted` rows included**); 3. `INSERT` only the absent rows — parameterised multi-row, at most **1,000 rows per statement**; 4. [wait for WAL visibility](#wal-visibility-and-the-read-your-writes-wait). Because `DEDUP UPSERT` would otherwise let a re-sent sample overwrite the stored one, step 2 is what keeps outbox redelivery idempotent — and **a live-writer redelivery never resurrects a deleted row**: a tombstone counts as present. Any SQL or transport failure returns `false` (the recorder retries; the #538 per-tag quarantine still applies); the writer never throws. ### WAL visibility and the read-your-writes wait QuestDB applies WAL commits asynchronously, so a row just inserted may not be visible to the next query (measured: up to 7 of 20 inserts invisible to an immediate re-read). **Every write path waits** — the live writer, the HistoryUpdate updater (including between DeleteRaw pages), the tag catalog/provisioner and the alarm writer (`QuestDbWalVisibility`): after the INSERT(s) it reads `sequencerTxn` from `wal_tables()` and polls `writerTxn` about every 10 ms until it catches up, the table is reported `suspended` (Warning, stop), or `VisibilityTimeout` elapses (Warning). A timeout never changes a result code and never discards data; the rows simply become visible shortly after. Measured: median 0 ms, p95 6 ms. ### HistoryUpdate (soft delete) All five operations are supported, and **every one is an `INSERT`**: QuestDB `UPDATE`/`DELETE` are never issued. "Exists" means a row that is present **and not deleted**. | Operation | Effect | Result | |---|---|---| | Insert | write the row when none exists — **a tombstone counts as absent, so Insert over a tombstone succeeds** | `GoodEntryInserted` / `BadEntryExists` | | Replace | overwrite (dedup upsert) when a live row exists | `GoodEntryReplaced` / `BadNoEntryExists` | | Update | write either way | `GoodEntryReplaced` or `GoodEntryInserted` | | DeleteAtTime | write a tombstone (`deleted = true`, value NULL, `BadNoData`) where a live row exists | `Good` / `BadNoEntryExists` | | DeleteRaw | tombstone every live row in `[start, end)`, 1,000 per page, waiting for visibility between pages | the same codes as `SqlServer` | **Modification trail (#668).** Each operation first appends one `{p}sample_modified` row per sample it will change, carrying the live row's current value (read by the same existence query), then writes the change. No multi-table transaction exists, so this order is the guarantee: a crash between the two leaves an **orphan trail row**, never an untracked change — the documented deviation from `SqlServer`'s atomic trail. See [Modified-value history](#modified-value-history-part-11). The per-item codes come from the shared `HistoryUpdatePlanner`, so they equal `SqlServer`'s (including the duplicate-timestamp-in-one-request rules above). **Every read and aggregate filters `NOT deleted`.** A client Insert at a deleted timestamp re-populates it (the OPC UA intent); only the *live writer's* redelivery is barred from doing so. ⚠️ **Concurrency.** The updater serializes calls **per tag, in-process only** (64 striped locks held from the existence read through the visibility wait), so two concurrent Insert/Replace/Delete batches on one tag cannot both act on a stale read. It does **not** lock against the live writer, nor against a second node. The existence checks read committed data, so a recorder sample at the same timestamp still in flight in the WAL can race an Insert/Replace (updates normally target past timestamps), and two nodes writing the same *new* key concurrently can produce last-write-wins (QuestDB has no `UPDLOCK`; the Primary gate makes it rare). This residual is accepted. ### Reads Every query filters `NOT deleted`; every value goes through `StoredSample`, so the quality clamp equals `SqlServer`'s. - **Raw:** `ts` range, `ORDER BY ts [DESC] LIMIT $n` (bound, `int`); `ContinuationPoint` is always null — the node manager pages by re-reading from the last timestamp ± `TimestampResolutionTicks`. - **AtTime:** previous-or-equal and next neighbour of each time by `LIMIT 1` seeks, `UNION ALL`-ed, **50 times per statement** (QuestDB 10.0.1 refuses more than 126 branches). `AtTimeEvaluator` decides each value, as on `SqlServer`. - **Events:** `[start, end)` by `ts`, ordered `(ts, event_id)`, `maxEvents + 1` fetched to produce the continuation point; a reverse read (#675) is `(lo, hi]` ordered `(ts, event_id)` descending. - **Processed:** one query per node; bucket index `(cast(ts AS LONG) * 10 - bucketBaseUnixTicks) / intervalTicks` (never `SAMPLE BY`, whose alignment differs), so bucket edges are identical to `SqlServer` (forward start-inclusive / end-exclusive with the partial bucket last; reverse anchored at the later time, later-edge-inclusive, partial bucket earliest — #677; interval 0 ⇒ one bucket, 100,000-bucket cap). On a per-sample read of any aggregate but Average, in either direction, two `LIMIT 1` statements fetch the tag's data range — its earliest live sample at or before the window's later edge and its latest at or after its earlier edge — for the start/end-of-data Partial bits and Count's in-range empty interval (#677/#680), read before the bucket statement. They seek the `tag` symbol index (#683; ~1–1.5 ms warm, 2–4 ms cold on a 53 M-row table whatever the tag's history — see *Data-range queries* above). Start/End/Delta also return each bucket's first and last sample of any quality (`row_number()` over every row). Min/max times use `row_number()` windows with the earliest-timestamp tie-break. ⚠️ **Live-gate 8 F1 — counts and picks must come from one table version.** QuestDB opens a separate table reader per reference to a CTE, so the earlier Start/End/Delta form (counts CTE + self-joins) could count a row committed between two references without finding it — a NULL first-of-any-quality time that threw and failed the node's read (~1 in 90k reads of a tag being written). That query now reads the table **once**: one `GROUP BY` with `first()`/`last()` for the first/last sample of any quality and `arg_min`/`arg_max` keyed on the usable rows' time for the first/last usable one (~15 % faster than before; the usable picks carry no `str_value` — `arg_min` has no VARCHAR form, and only Delta reads them, never with a usable string in the bucket). On `SqlServer` (READ COMMITTED gives no cross-reference guarantee either, though it was never observed there) Start/End/Delta was first reshaped into a windowed pass + conditional `MAX`; **#683 replaced that** — its four `ROW_NUMBER` sorts spilled and it measured *slower* than the five-reference form on long windows. It is now **aggregate, then join back on the key**: one `GROUP BY` computes the counts and each bucket's four pick *times* (`MIN`/`MAX(SourceTimeUtc)`, any quality and usable), and each pick's columns are fetched by a seek on the clustered key `(TagId, SourceTimeUtc)` — **forced** with `WITH (FORCESEEK (PK_Sample (TagId, SourceTimeUtc)))`, so every pick is a nested-loops seek on both key columns, one row per bucket per pick, whatever the statistics say. The optimizer's own choices were both pathological, and plans are cached and reused across reads and tags: keyed on the pick time alone, a hash/merge join whose inner input was the tag's **whole history** (#683 review: 43,657 logical reads vs 361 for a 1 d @ 1 h read of a tag with 3 M rows outside the window); bounded to the window as well, a nested-loops **range** seek over the window with the pick time as a residual, once per bucket — O(buckets × window rows) — compiled whenever the statistics understate the table (a fresh or freshly bulk-loaded historian, ascending-key estimates for recent windows) and then reused for a wide read (**live gate 9 F1**: 1 d @ 1 min 20 s). With the hint neither shape exists: a forced `HASH`/`MERGE JOIN` is refused (error 8622). ⚠️ The hint names the primary key `PK_Sample` created by `001_initial.sql`; a hand-built schema must keep that name. Output is identical to every earlier form (`EXCEPT` both ways empty, forward and reverse, including a tag of mixed Double/Int64/ Boolean/String/DateTime/outage samples). Measured on 6 M rows / 30 days for one tag (a 1 d or 7 d window leaves up to 5.8 M rows of that tag outside it) plus an 86 k-row mixed-kinds tag; elapsed ms / session logical reads, each variant's plan compiled on its first read and reused by the rest: | Window | before #683 (windowed) | #683 as merged, accurate stats | #683 as merged, understated stats ‡ | **now (FORCESEEK)**, accurate | **now**, understated ‡ | |---|---|---|---|---|---| | 1 d @ 1 min (1440 buckets) | 329 / 6.7 k | 121 / 14.9 k | **47,223 / 3.6 M** | 160 / 25.0 k | 253 / 25.6 k | | 1 d @ 1 min, reverse | 371 / 6.6 k | 129 / 14.9 k | **46,806 / 3.6 M** | 167 / 25.0 k | 246 / 25.6 k | | 1 d @ 1 h (24 buckets) | 578 / 5.4 k | 79 / 5.5 k | 984 / 63 k | 117 / 1.8 k | 206 / 3.6 k | | 7 d @ 1 min (10,080 buckets) | 3,426 / 103 k | 1,019 / 92 k | (hours — not run) | 1,472 / 180 k | 1,819 / 179 k | | 30 d @ 1 h (720 buckets) † | 35,339 / 322 k | 1,487 / 191 k | (hours — not run) | 5,720 / 73 k | 6,921 / 101 k | | mixed kinds 1 d @ 1 min | 190 / 5.9 k | 92 / 12.0 k | **20,164 / 1.66 M** | 107 / 24.4 k | 137 / 24.0 k | | mixed kinds 1 d @ 7 min, reverse | 199 / 1.5 k | 71 / 3.9 k | 2,977 / 238 k | 59 / 4.2 k | 93 / 4.7 k | ‡ `UPDATE STATISTICS … WITH ROWCOUNT = 34, PAGECOUNT = 1`, then the plan compiled on a 1 h @ 1 min read and reused — gate 9's recipe. † The 30 d row is dominated by the window `GROUP BY`, not the picks: compiled on the 30 d read itself, the merged query and the hinted one both take ~6.9 s (serial, a small memory grant, spills); the merged query's 1.5 s above is a plan compiled on the 1 d read and reused. Alternatives measured and rejected: the same seek written as `OUTER APPLY (SELECT TOP (1) … WITH (FORCESEEK …))` (identical plans and numbers — the join form is shorter); `OPTION (RECOMPILE)` on the merged query (compiles under the bad statistics on every call: still 20.4 s / 1.66 M reads on the mixed-kinds tag, and a compile per read). `StartEndJoinBackTests` pins it — one sample a second over a day with 300 k rows of history either side: under understated statistics 18,645 reads (the merged query: **1,645,845 reads, 23 s**, so the test failed against it), with accurate statistics 22,616, and forced `HASH`/`MERGE` joins refused with 8622. (The dev SQL container is slow in absolute terms — the ratios are the point.) ⚠️ The seeks are separate reads of the table within one statement, so a sample **deleted** by HistoryUpdate between the aggregate and its seek leaves that pick NULL while the counts include it: the null-tolerant reader below answers that bucket's Start/End/Delta `BadNoData`, transiently. A usable pick's join also re-checks usability (the exact `u` predicate), so a HistoryUpdate **Replace** that turned the picked sample Bad/Empty in between reads the same way — never a Bad row's value under Delta. Min/MaxActualTime keep their counts + filtered-rank form on both backends: every single-scan variant measured 1.2–2.4× slower, and QuestDB's `arg_max` cannot express the earliest-time tie-break. So, on both backends, a row whose picks are absent anyway maps to "no such sample" — `BadNoData` for that interval, never an exception and never a value-less Good. `LiveWriteReadRaceStressTests` (QuestDB integration, opt-in via `HISTORIAN_QUESTDB_STRESS_SECONDS`) interleaves a single-row writer with tight-loop reads to pin it. ### Retention (TTL) `RetentionDays` becomes `TTL n DAYS` on `{p}sample`, `{p}event` and `{p}sample_modified` (never on `{p}tag`: TTL must be a multiple of the partition size). QuestDB drops expired day partitions itself, tombstones included. - At bootstrap the current TTL is read from `tables()` (`ttlValue`, `ttlUnit`). QuestDB **normalises the unit** (7 days reads `1 WEEK`), so the bootstrapper compares in **hours** (`HOUR`=1, `DAY`=24, `WEEK`=168; any other unit counts as different) and issues `ALTER TABLE … SET TTL n DAYS` only on a difference (`SET TTL 0 DAYS` removes it). The `ALTER` is applied through the WAL, so it does not re-read to verify; a very large `RetentionDays` that QuestDB normalises into MONTH/YEAR is re-`ALTER`ed on every boot (harmless). - ⚠️ **TTL is enforced only on write commits.** QuestDB checks for expired partitions when a write commits to the table, so **a quiet table never drops anything**: partitions older than the TTL linger until the next write lands. Do not read a stale-looking partition as a broken TTL. - **Expiry is judged against the wall clock** (`min(max timestamp, now)`), so a future-dated sample does not expire current data. ⚠️ **Keep `cairo.ttl.use.wall.clock` at its default, `true`** (#674). It is what QuestDB documents as protecting a TTL table from a future-dated row; with it `false` the TTL would be measured from the table's newest timestamp and one far-future row (a 2062 typo) could drop every older partition — including `{p}sample_modified`, the record of who changed history. Evidence (QuestDB 10.0.1, 2026-10-11): the default is `true`, and a 2066 row inserted into a 30-day-TTL table evicted nothing inside the TTL, while a partition genuinely older than 30 days was evicted by that same commit — pinned by `TtlWallClockTests`. (The same run with the flag `false` also kept the in-TTL partitions, so the test pins the outcome rather than isolating the flag; do not rely on that.) The server-side `ServerHistorian:HistoryUpdateMaxFutureSkew` bound (default 1 day) is the primary guard: a HistoryUpdate cannot write a far-future row. - The TTL is visible in table metadata: `SELECT table_name, ttlValue, ttlUnit FROM tables() WHERE starts_with(table_name, 'hist_')`. ### Docker-dev recipe QuestDB is an **optional compose overlay**, `docker-dev/docker-compose.questdb.yml`, so the default rig is unchanged. See [`docker-dev/README.md`](../docker-dev/README.md#questdb-historian-overlay). ```bash # 1. Bring up ONLY the questdb service (PG wire on host :18812, console/REST on host :19000). docker compose -f docker-dev/docker-compose.yml -f docker-dev/docker-compose.questdb.yml up -d --no-deps questdb # 2. Recreate the central nodes against it (the QuestDb connection string already defaults to the `questdb` service): OTOPCUA_HISTORIAN_ENABLED=true OTOPCUA_HISTORIAN_BACKEND=QuestDb OTOPCUA_CONTHIST_ENABLED=true \ docker compose -f docker-dev/docker-compose.yml -f docker-dev/docker-compose.questdb.yml up -d central-1 central-2 # 3. Mark a rig tag historized, deploy, then watch rows land (REST, no client needed): curl -G localhost:19000/exec --data-urlencode "query=SELECT tag, count() n, max(ts) last FROM hist_sample WHERE NOT deleted GROUP BY tag" ``` Only the Primary of a pair records. Read back with `Client.CLI historyread`, rewrite with `Client.CLI historyupdate`. ### Integration tests `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests` (bootstrap incl. two concurrent bootstrappers, TTL via `tables()`, writes, tombstones and redelivery, the visibility wait, updater, reads, paging, and the shared conformance suite) is gated on **`HISTORIAN_QUESTDB_TEST_CONNECTION`** — an Npgsql connection string for the PG-wire port. Absent ⇒ every test skips. Each test class uses a unique throwaway prefix (`t__`) and drops its tables afterwards. The QuestDB server allows 64 connections, so the fixture prunes idle pool connections. ```bash docker compose -f docker-dev/docker-compose.yml -f docker-dev/docker-compose.questdb.yml up -d --no-deps questdb export HISTORIAN_QUESTDB_TEST_CONNECTION='Host=localhost;Port=18812;Username=admin;Password=quest;Database=qdb;ServerCompatibilityMode=NoTypeLoading' dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests ``` --- ## HistoryUpdate OPC UA **HistoryUpdate** lets a client insert, replace, update and delete stored raw history. It is available only when the selected backend provides an updater — **`SqlServer` and `QuestDb` do; `Gateway` does not**, and an updater is registered only when `ServerHistorian:Enabled=true`. Updates go straight to the backend (not through the outbox), are **Primary-only** (#674, [below](#primary-only-674)), and share the HistoryRead batch limiter and `HistoryReadDeadline`. ### Operations | OPC UA request | Server override | Per-item results (`OperationResults`) | |---|---|---| | `UpdateDataDetails`, `PerformInsertReplace = Insert` | `HistoryUpdateData` | `GoodEntryInserted`; `BadEntryExists` if a sample already exists at that timestamp | | …`Replace` | `HistoryUpdateData` | `GoodEntryReplaced`; `BadNoEntryExists` if none exists | | …`Update` | `HistoryUpdateData` | `GoodEntryInserted` or `GoodEntryReplaced` (upsert) | | `DeleteRawModifiedDetails` (`IsDeleteModified=false`) | `HistoryDeleteRawModified` | one node-level code: `Good`; `BadNoData` if `[start, end)` held nothing | | `DeleteAtTimeDetails` | `HistoryDeleteAtTime` | per time: `Good`; `BadNoEntryExists` | Each backend call is applied as a unit (SqlServer: one transaction, `MERGE … OUTPUT $action` against the TVP; QuestDb: an INSERT per chunk plus the WAL-visibility wait, no transaction — see [QuestDB backend](#questdb-backend)). Every applied item is also recorded, with the caller's principal, in the [modification trail](#modified-value-history-part-11) (#668). The node-level `StatusCode` stays `Good` when item results are mixed (Part 11 reports item outcomes in `OperationResults`); it is Bad only for whole-node failures. **Per-item rejections** (answered without touching SQL; other items still proceed): a null item ⇒ `BadInvalidArgument`; `SourceTimestamp == DateTime.MinValue` ⇒ `BadInvalidTimestamp`; a **`SourceTimestamp` later than `now + ServerHistorian:HistoryUpdateMaxFutureSkew`** (default 1 day, server clock; #674 — a `Local`-kind timestamp is converted to UTC first, and is sent as UTC) ⇒ **`BadOutOfRange`** — on every backend, never sent; a value the historian cannot type (array, unsupported CLR type) ⇒ `BadTypeMismatch`; a **`NaN` / `±Infinity` `Double` ⇒ `BadInvalidArgument`** — it cannot live in a SQL `float`, and unlike the live-value writer (which stores it as `Empty`) an operator's explicit update is rejected rather than silently rewritten; and a **null value with a Good or Uncertain status ⇒ `BadInvalidArgument`** — an `Empty` sample is always Bad, so a null value is accepted only with a Bad status (an operator-authored outage marker). A caller-supplied status code is otherwise stored as given. **No DataType check.** HistoryUpdate does **not** compare a value's type with the node's `DataType`: a value is stored as sent (a `String` written to a `Double` node's history is stored and later read back as a `String`), exactly as the live recorder stores whatever type the driver publishes. Only a value the historian cannot represent at all is refused (`BadTypeMismatch`, above). Part 11 §6.8 permits a server to answer `BadTypeMismatch` for a value of the wrong type; this server deliberately does not. **`DeleteRawModifiedDetails` needs both bounds:** a `StartTime` or `EndTime` of `DateTime.MinValue` (the wire's "unspecified") is `BadInvalidArgument` for that node and never reaches the backend. **Duplicate timestamps inside one request** (compared at 100 ns): *Insert* — the first occurrence is applied, later ones are `BadEntryExists`; *Replace* / *Update* — the last one wins and every earlier occurrence reports the winner's code; *DeleteAtTime* — the first reports the real outcome, later ones `BadNoEntryExists`. **Unknown tag:** Insert / Update create the tag row; Replace and the deletes report `BadNoEntryExists` (DeleteRaw: `BadNoData`) without creating anything. **Unsupported ⇒ `BadHistoryOperationUnsupported`** (per node): the node is not historized; the backend has no updater (`Gateway`, or `ServerHistorian:Enabled=false`); `PerformUpdateType.Remove`; `DeleteRawModifiedDetails.IsDeleteModified=true` (the [modification trail](#modified-value-history-part-11) is an audit record, aged out only by retention); and the out-of-scope detail types — **event** HistoryUpdate (`UpdateEventDetails` / `DeleteEventDetails`) and `UpdateStructureData` — which are not overridden, so the SDK base rejects them. A backend exception is `BadInternalError` for that node and a deadline is `BadTimeout`. On a `SqlServer` database still at schema v1 the whole node is `BadHistoryOperationUnsupported` on every arm ([#674](#modified-value-history-part-11)). No `AuditHistoryUpdateEvent` is emitted (#674.5, not implemented). ### Primary-only (#674) HistoryUpdate is served only by the node holding the redundancy **Primary** data-plane role — the same verdict (`IDriverPrimaryRoleSource`, published by `DriverHostActor`) that gates native-alarm acknowledges (#597). On any other node every node of the request that **passed authorization and is servable** is refused **`BadInvalidState`** (node-level code, with a message pointing at the Primary) and nothing reaches the backend. Order: authorization first (an unauthorized caller learns `BadUserAccessDenied`, not the node's role), then the request is **planned** (a request no server could serve — no updater, a node that is not historized, `Remove`, `IsDeleteModified` — is the honest `BadHistoryOperationUnsupported` on every node, so a Secondary does not send the client to a Primary that would refuse it too), then the gate. **Closed until the first redundancy snapshot (boot window).** The gate reads its own verdict, `IHistoryUpdateRoleView.ShouldServeHistoryUpdate` (`PrimaryGatePolicy.ShouldServeHistoryUpdate`): the data-plane Primary verdict, but **closed until this node has received its first redundancy snapshot**. The data-plane verdict itself allows during the pre-join boot window (a node not yet joined sees no driver peer, which reads as "single node") — right for live writes and acks, which must not stall on a genuine single node, but it would let both nodes of a pair accept a HistoryUpdate for a few seconds (up to ~25 s under the bootstrap guard). "No verdict yet" is told apart from "single node" by the snapshot itself: the redundancy singleton runs on every driver node — a lone node included — and publishes on every change and every 10 s heartbeat, so a single node is named Primary within seconds of joining and opens then. Until that first snapshot every node refuses `BadInvalidState`; after it the gate is exactly the data-plane verdict (including the unknown-role / member-count rule for a snapshot that omits this node). A node with no redundancy manager (admin-only — it registers no `IHistoryUpdateRoleView`) wires no gate and allows. HistoryRead is **not** gated. Why: rewriting history is a single-writer operation — both backends capture the prior value for the [modification trail](#modified-value-history-part-11) under a per-tag, **per-process** lock (QuestDB) or statement (SqlServer), so two pair nodes updating one tag concurrently could record a wrong prior value. A single-node deployment (no redundancy manager) is unaffected. Clients should send HistoryUpdate to the server advertising the higher `ServiceLevel`. ### The `HistoryWrite` AccessLevel bit The SDK base admits a node to HistoryUpdate only if its `AccessLevel` carries `HistoryWrite`. The server ORs that bit in **only when the node is historized *and* an updater is wired at materialise time** (the raw node and every UNS mirror; the surgical `UpdateTagAttributes` path recomputes it the same way). `OtOpcUaServerHostedService.StartAsync` therefore wires the updater **before** installing the sink, and nodes materialised earlier are never retro-updated. With `Backend=Gateway` the bit is never set. ### Authorization HistoryUpdate is gated in **both** ACL modes (rewriting recorded history was never ungated). `WriteOperate` is deliberately **not** reused. | `Security:Acl:Enabled` | Decision | |---|---| | `false` (default) | The session's API key is re-checked first (`AuthorizeKeyUsable`, Gitea #563 — an expired or disabled key loses its activation-time roles), then the session must carry the **`HistoryUpdate` role**. `ApiKeyInterimRoleMapper` confers it **only from a `Cluster`-scoped grant** carrying `NodePermissions.HistoryUpdate`; narrower-scoped grants confer nothing (the same under-grant rule as `WriteOperate`). Denied ⇒ `BadUserAccessDenied`. A `WriteOperate`-only key is denied. | | `true` | `DataPlaneAuthorizer.Authorize(identity, OpcUaOperation.HistoryUpdate, nodeId, realm)` — the scoped `NodeAcl` grant decides per node and **replaces** the role check (never chained), so a session with no roles succeeds on the strength of a scoped grant. Denied ⇒ `BadUserAccessDenied` for that node. | `NodePermissions.HistoryUpdate` (bit 12) is authorable in the AdminUI ACL editor and is **not** part of any bundle (`ReadOnly` / `Operator` / `Engineer` / `Admin`) — it must be granted explicitly. See [security.md](security.md#data-plane-authorization). --- ## Authoring workflow 1. Open the equipment's **Tags** tab on `/uns/equipment/{id}`. 2. Create or edit the tag. - For **typed-editor drivers** (Modbus, S7, OpcUaClient, etc.): check the **Historize this tag** checkbox and, if needed, fill in the **Historian tagname (override)** textbox. - For **raw-JSON editors** (Galaxy): you can check the same first-class checkboxes (they appear below the JSON textarea), or add `"isHistorized":true` (and optionally `"historianTagname":"..."`) directly in the textarea. 3. Save and publish. The server rebuilds its address space; the node materialises with `Historizing=true` and the `HistoryRead` AccessLevel bit. 4. Confirm with Client.CLI `read` that the node's `Status` is `Good` and that the value is updating. Then issue a `historyread` to verify the historian connection returns data. ### Native-alarm historian opt-out (`alarm.historizeToAveva`) A tag carrying a native `"alarm"` object has a **separate** historian opt-out for its **alarm transitions** (distinct from tag-value historization). On the Tag modal, check or uncheck **Historize to AVEVA** in the alarm section. This maps to `alarm.historizeToAveva` (`bool?`) in the `TagConfig` JSON: - **Absent or `true`** (default) — the alarm's transitions are written to AVEVA Historian via `HistorianAdapterActor`. - **`false`** — the durable AVEVA write is suppressed for this alarm's transitions. The live `/alerts` feed and OPC UA condition events are unaffected. The gate is applied in `HistorianAdapterActor` using `is not false` semantics, matching the scripted-alarm `HistorizeToAveva` posture. See [ScriptedAlarms.md §Native driver alarms](ScriptedAlarms.md#native-driver-alarms-equipment-tag-path) and [AlarmTracking.md §Historian write-back](AlarmTracking.md#historian-write-back-non-galaxy-alarms) for the full alarm-historian routing. --- ## Client.CLI historyread examples The `historyread` command reads historical data from any node. Supply start and end times in ISO 8601 UTC form. See [docs/Client.CLI.md](Client.CLI.md) for the full flag reference. > **v3 Batch 4 NodeIds.** A historized tag is readable through **either** of its two NodeIds — > the Raw-namespace node (`s=`, e.g. `Plant/Modbus/dev1/Speed`) or a referencing > equipment's UNS-namespace node (`s=///`). Both register > the same historian tagname, so the returned series is identical. The `ns=N` index is assigned > at connect time per the namespace URI (`https://zb.com/otopcua/raw` / > `https://zb.com/otopcua/uns`) — resolve it from the server's `NamespaceArray` rather than > hard-coding it. The examples below show a Raw-namespace read (`ns=2` illustrative). ```bash # Raw history for a historized tag via its Raw-namespace NodeId (last 24 hours by default) otopcua-cli historyread \ -u opc.tcp://localhost:4840/OtOpcUa \ -n "ns=2;s=Plant/Modbus/dev1/Speed" \ --start "2026-06-13T00:00:00Z" --end "2026-06-14T00:00:00Z" # Same series via a referencing equipment's UNS-namespace NodeId, limited to 100 values otopcua-cli historyread \ -u opc.tcp://localhost:4840/OtOpcUa \ -n "ns=3;s=filling/line1/station1/Speed" \ --start "2026-06-13T00:00:00Z" --end "2026-06-14T00:00:00Z" \ --max 100 # 1-hour average aggregate otopcua-cli historyread \ -u opc.tcp://localhost:4840/OtOpcUa \ -n "ns=2;s=Plant/Modbus/dev1/Speed" \ --start "2026-06-13T00:00:00Z" --end "2026-06-14T00:00:00Z" \ --aggregate Average --interval 3600000 # Authenticated read (ReadOnly role or higher required) otopcua-cli historyread \ -u opc.tcp://localhost:4840/OtOpcUa \ -n "ns=2;s=Plant/Modbus/dev1/Speed" \ --start "2026-06-13T00:00:00Z" --end "2026-06-14T00:00:00Z" \ -U reader -P password ``` Supported `--aggregate` values: `Average`, `Minimum`, `Maximum`, `Count`, `Start`, `End`, `StandardDeviation` (aliases: `avg`, `min`, `max`, `stddev`/`stdev`, `first`, `last`). `--interval` is the processing interval in milliseconds (default 3600000 = 1 hour). The `Total` aggregate is served by the server's OPC UA HistoryRead endpoint for any conformant client (derived as time-weighted Average × interval-seconds — see "Total aggregate derivation" above), but is not exposed by this bundled CLI. --- ## Live /run gate The live read gate requires a reachable `ZB.MOM.WW.HistorianGateway` (VPN to `wonder-sql-vd03`) with the AVEVA Historian behind it healthy. Set `ServerHistorian:Enabled=true` with the correct `Endpoint` (`https://host:5222`) and supply `ServerHistorian__ApiKey` via the environment, then deploy and publish at least one historized Galaxy tag. The gate is operator-driven — it is not part of the local docker-dev rig. The gateway-backed driver also ships an env-gated live suite (`Category=LiveIntegration`); see the `HISTGW_GATEWAY_ENDPOINT` / `HISTGW_GATEWAY_APIKEY` / `HISTGW_TEST_TAG` / `HISTGW_WRITE_SANDBOX_TAG` / `HISTGW_BOOL_SANDBOX_TAG` / `HISTGW_ALARM_SOURCE` env vars (it skips cleanly when they are absent). `HISTGW_BOOL_SANDBOX_TAG` (e.g. `HistGW.LiveTest.BoolSandbox`, a **dedicated** writable Boolean tag — never the shared Float write sandbox) drives the Boolean historization leg (`EnsureTags_boolean_sandbox_provisions_as_Int2`) and the retype probe (`EnsureTags_type_change_outcome_is_observable`) that back the [Boolean historization maps to Int2](#boolean-historization-maps-to-int2-issue-441) decision. Running the full suite against the gateway (all six tests, none skipped) also discharges 06/U-2's "full documented run" for the historian suite. See [AlarmHistorian.md](AlarmHistorian.md) for the alarm write path and [ServiceHosting.md](ServiceHosting.md) for the (external) HistorianGateway deployment. --- --- ## Cross-driver TagConfig key reference The `TagConfig` JSON blob is the single extension surface for per-tag server-side behaviours across all drivers. Keys from multiple phases coexist in the same blob; unknown keys are preserved byte-stable on round-trip by all typed editors. ### Historian + alarm keys (Phase B / Phase C) | Key | Type | Phase | Description | |---|---|---|---| | `isHistorized` | bool | C | Marks the tag historized (HistoryRead + Historizing=true). | | `historianTagname` | string | C | Explicit historian tagname override (defaults to `FullName`). | | `alarm` | object | B | Native alarm definition (`alarmType`, `severity`, `historizeToAveva`, …). | ### Array keys (Phase 4c) | Key | Type | Phase | Description | |---|---|---|---| | `isArray` | bool | 4c | When `true` and `arrayLength >= 1`, materialises the node as a 1-D array (`ValueRank=OneDimension`). Absent or `false` → scalar. | | `arrayLength` | uint (≥1) | 4c | Element count; sets `ArrayDimensions`. Required when `isArray: true`. | ### Cross-driver array coverage matrix | Driver | Read mechanism | Live-verify status | |---|---|---| | **Modbus** | Contiguous FC03/FC04 block; String/BitInRegister modes supported | Mac-verifiable (sim `10.100.0.35:5020`) | | **S7** | `ReadBytesAsync` block + per-element decode loop | Unit-proven (fixture down) | | **AB CIP** | libplctag native array read (atomic + UDT member arrays) | Unit-proven (fixture down) | | **AB Legacy** | PCCC multi-element file read via libplctag (cap 256 elements) | Unit-proven (fixture down) | | **TwinCAT** | Raw-bytes array read + per-element decode (native notifications carry the whole array) | Live-verified on TwinCAT 2.11 (2026-10-07) | Array writes, multi-dimensional arrays (ValueRank>1), and per-element historization are out of scope — see [Uns.md §Array tags](Uns.md#array-tags-1-d). --- ## Closed backlog §2 items (Phase 4 / Phase 4b) The following items from the backlog §2 (archived at `docs/plans/2026-06-18-stillpending-backlog-audit.md`) were closed by earlier phases and are recorded here so future audits don't re-flag them. | Item | Closed by | Commit | |---|---|---| | Modbus Int64 / UInt64 OPC UA node `DataType` correction | Phase 4 | `bd8fee61` | | `HistoryAggregateType.Total` — `Total` aggregate | Phase 4 (derived client-side as time-weighted Average × interval-seconds) | `5e27b5f7` | | Historian poison alarm-event indefinite retry — dead-letter cap (task #437) | Phase 4 | `fcb38014` | --- ## See also - [docs/plans/2026-06-14-galaxy-phase-c-historian-design.md](plans/2026-06-14-galaxy-phase-c-historian-design.md) — full design and implementation notes - [AlarmHistorian.md](AlarmHistorian.md) — alarm write path; drains to the same HistorianGateway (`SendEvent`) - [AlarmTracking.md](AlarmTracking.md) — OPC UA Part 9 alarm surface (event history source) - [Client.CLI.md](Client.CLI.md) — full `historyread` flag reference - [ScriptedAlarms.md](ScriptedAlarms.md) §"Native driver alarms" — the Phase B `alarm` object in `TagConfig` (parallel carrier) - [Uns.md §Array tags](Uns.md#array-tags-1-d) — `isArray`/`arrayLength` keys, cross-driver coverage, and deferrals