# Historian — server-side OPC UA HistoryRead / HistoryUpdate (pluggable backends) Phase C wires server-side OPC UA **HistoryRead** for authored equipment tags flagged historized. The feature is driver-agnostic: any equipment tag (Galaxy, Modbus, OpcUaClient, or any other driver) can be marked historized; the server dispatches all history reads to the registered `IHistorianDataSource` — the **HistorianGateway** read client (`GatewayHistorianDataSource`, talking gRPC to the external `ZB.MOM.WW.HistorianGateway` via the `ZB.MOM.WW.HistorianGateway.Client` package). No EF migration is required; the historian flag rides in the existing schemaless `TagConfig` JSON blob alongside the Phase B `alarm` object. (The bespoke Wonderware TCP sidecar backend this replaced was retired — see [drivers/Historian.Wonderware.md](drivers/Historian.Wonderware.md).) > **The backend is pluggable (2026-10).** `ServerHistorian:Backend` selects `Gateway` (the default — everything > described below as "the gateway") or `SqlServer` (a plain SQL Server database the server reads and writes > itself, which also supports OPC UA **HistoryUpdate**). See [Backend selection](#backend-selection-serverhistorianbackend), > [SQL Server backend](#sql-server-backend) and [HistoryUpdate](#historyupdate). Wherever this page says > "the historian" without qualification it means the selected backend; sections that only apply to one backend say so. Design references: [docs/plans/2026-06-14-galaxy-phase-c-historian-design.md](plans/2026-06-14-galaxy-phase-c-historian-design.md) (Phase C) and [docs/plans/2026-10-09-pluggable-historian-design.md](plans/2026-10-09-pluggable-historian-design.md) (pluggable backends). --- ## Historized TagConfig schema A tag is historized by setting the **Historize this tag** checkbox and optional **Historian tagname (override)** textbox in the Tag modal on the `/uns` equipment page Tags tab. These controls work for **all drivers** — typed editors (Modbus, S7, OpcUaClient, etc.) and the raw-JSON textarea (Galaxy) alike. The controls merge `isHistorized` / `historianTagname` into the existing `TagConfig` JSON blob via the `TagHistorizeConfig` helper, preserving all other keys byte-stable. Drivers that still use the raw-JSON editor (Galaxy) can also add the fields directly in the textarea. ### Fields | Field | Type | Required | Description | |---|---|---|---| | `isHistorized` | bool | yes | Marks the tag historized. Materialises the OPC UA node with `Historizing=true` and the `HistoryRead` AccessLevel bit. | | `historianTagname` | string | no | Explicit historian tagname to query. When absent or empty, defaults to the tag's driver `FullName` (the `TagConfig.FullName`). | ### Examples Default historian tagname (uses `FullName` = `TestMachine_002.TestFloat`): ```json {"FullName":"TestMachine_002.TestFloat","isHistorized":true} ``` Explicit historian tagname override: ```json {"FullName":"TestMachine_002.TestFloat","isHistorized":true,"historianTagname":"Plant.Line1.Flow"} ``` A tag that is both historized and a native alarm: ```json {"FullName":"TestMachine_002.HiAlarm","isHistorized":true,"alarm":{"alarmType":"OffNormalAlarm","severity":700}} ``` --- ## Backend selection (`ServerHistorian:Backend`) `ServerHistorian:Backend` (case-insensitive, trimmed; default `Gateway`) picks the provider (`IHistorianBackendProvider`) that builds all five historian seams — the HistoryRead data source, the continuous- historization value writer, the alarm-history writer, the tag provisioner and (optionally) the HistoryUpdate updater. `HistorianBackendRegistry` in the Host owns the name table. An unknown name fails host start with an `OptionsValidationException` that lists the allowed values (`Gateway`, `SqlServer`), regardless of whether anything consumes the historian. The existing enable gates (`ServerHistorian:Enabled`, `AlarmHistorian:Enabled`, `ContinuousHistorization:Enabled`) are unchanged — they simply get their objects from the selected provider. | | `Gateway` (default) | `SqlServer` | |---|---|---| | Storage | AVEVA Historian via the HistorianGateway sidecar | `{Schema}` tables in a SQL Server database you point it at | | Value kinds recorded | `Double`, `Int64`, `Boolean` (numeric analog only) | all: `Double`, `Int64`, `Boolean`, `String`, `DateTime`, plus `Empty` (a Bad sample with no value) | | Status code stored | Sent as DA quality but **discarded** by the gateway (C-002); AVEVA stores every sample Good | Stored bit-for-bit (`StatusCode` int column) | | Comms-loss / driver-Bad publishes | Dropped, counted in `DroppedUnsupportedKind` | Stored as an `Empty` outage-marker sample carrying the Bad code (see [below](#typed-values-real-status-and-outage-markers)) | | Processed aggregates | Average, Minimum, Maximum, Total, Count | The thirteen in the [aggregate table](#aggregates-processed-reads) | | HistoryUpdate | No (`BadHistoryOperationUnsupported`) | Yes (Insert / Replace / Update / DeleteRaw / DeleteAtTime) | | Alarm history | gateway `SendEvent` | `Event` table (insert-if-absent) | | Provisioning (`EnsureTags`) | numeric types only; others skipped | `MERGE` upsert of every request, all types | | Connection | `Endpoint` + `ApiKey` + TLS keys (top-level `ServerHistorian:*`) | `ServerHistorian:SqlServer:*` | Recording stays **Primary-only** (#537) and **at-least-once** with the #538 per-tag quarantine on both backends; HistoryRead and HistoryUpdate are served from any node (the database is shared). The historian connection string is **not** a ConfigDb string, and a driver-only node may carry it without breaking the Phase 4 "no ConfigDb" posture. --- ## ServerHistorian configuration The `ServerHistorian` section in `appsettings.json` controls the historian read path. `Enabled` defaults to `false`; when disabled, the server registers `NullHistorianDataSource` and all HistoryRead calls on historized nodes return `GoodNoData` (empty, not an error). ```json { "ServerHistorian": { "Enabled": false, "Backend": "Gateway", "Endpoint": "", "ApiKey": "", "UseTls": true, "AllowUntrustedServerCertificate": false, "CaCertificatePath": null, "CallTimeout": "00:00:30", "MaxTieClusterOverfetch": 65536, "HistoryReadBatchParallelism": 4, "MaxConcurrentHistoryReadBatches": 16, "HistoryReadDeadline": "00:01:00", "SqlServer": { "ConnectionString": "", "Schema": "hist", "CommandTimeout": "00:00:30", "AutoCreateSchema": true } } } ``` | Key | Type | Default | Description | |---|---|---|---| | `Enabled` | bool | `false` | Enable the selected backend's read client. `false` → `NullHistorianDataSource` (empty reads) and no HistoryUpdate. | | `Backend` | string | `Gateway` | `Gateway` or `SqlServer` (case-insensitive). Unknown ⇒ host start fails. See [Backend selection](#backend-selection-serverhistorianbackend). | | `Endpoint` | string | `""` | **Gateway backend only.** Absolute gateway URI, e.g. `https://host:5222`. Scheme selects transport (`https://` = TLS, `http://` = h2c plaintext). Required when `Enabled`. | | `ApiKey` | string | `""` | **Gateway backend only.** The gateway peppered-HMAC key (`histgw__`) sent as `Authorization: Bearer`. Required when `Enabled`. **Supply via env `ServerHistorian__ApiKey`.** | | `UseTls` | bool | `true` | **Gateway backend only.** Connect over TLS; must match the `Endpoint` scheme. | | `AllowUntrustedServerCertificate` | bool | `false` | **Gateway backend only.** Accept a self-signed / untrusted server certificate (dev / on-prem only). | | `CaCertificatePath` | string\|null | `null` | **Gateway backend only.** PEM CA file pinning the gateway's TLS chain. Null/empty uses the OS trust store. | | `CallTimeout` | TimeSpan | `00:00:30` | **Gateway backend only.** Per-call deadline applied to each unary gateway read. A read for a tag the historian does not yet know about — e.g. a freshly-historized tag that has **not** been provisioned (see [Tag auto-provisioning](#tag-auto-provisioning-ensuretags)) — can run to this **full** deadline before the gateway errors, so an unprovisioned-tag HistoryRead can block for ~30 s. Lower this to fail faster, at the cost of truncating legitimately long reads. | | `MaxTieClusterOverfetch` | int | `65536` | (Node-manager paging; all backends — a `SqlServer` PK rules out tie clusters, so it is moot there.) Maximum samples the server will fetch in one shot to page through a tie cluster (multiple samples sharing one `SourceTimestamp`). A cluster larger than this ceiling fails `BadHistoryOperationUnsupported`. Raise to handle abnormally large tie clusters; the default covers all normal-data cases. | | `HistoryReadBatchParallelism` | int | `4` | Max historized nodes served **concurrently** within one HistoryRead batch (arch-review 03/S3). The SDK's `HistoryRead*` override surface is synchronous, so each arm bridges once at its boundary; this turns per-node serving from sequential (`N × CallTimeout` worst case) into `⌈N / P⌉ × CallTimeout`. Also caps the gateway-load multiplication a single client can command. `≤ 1` serves sequentially. | | `MaxConcurrentHistoryReadBatches` | int | `16` | Process-wide limit on HistoryRead (and HistoryUpdate) batches served **concurrently** (arch-review 03/S3). A flood of history clients degrades to `BadTooManyOperations` rather than exhausting the SDK request-thread pool. With `HistoryReadBatchParallelism` this caps total in-flight gateway reads at `MaxConcurrentHistoryReadBatches × HistoryReadBatchParallelism` (default 16 × 4 = 64). A batch that cannot get a slot within `min(HistoryReadDeadline, 5 s)` fails every handle `BadTooManyOperations`. | | `HistoryReadDeadline` | TimeSpan | `00:01:00` | Per-request wall-clock deadline for a HistoryRead (and HistoryUpdate) batch (arch-review 03/S3). A node still in flight at the deadline surfaces `BadTimeout`, so one request can never hold an SDK request thread longer than this regardless of node count. Non-positive = unbounded (a `Validate()` warning). | **`ServerHistorian:SqlServer`** (read only when `Backend=SqlServer`; bound by `HistorianBackendRegistry.BindSqlServer`): | Key | Type | Default | Validation | Description | |---|---|---|---|---| | `SqlServer:ConnectionString` | string | `""` | **Required** (non-blank) | ADO.NET connection string for the historian database. Never the ConfigDb. Supply via `${secret:OT-HISTORIAN-SQL}` / environment (`ServerHistorian__SqlServer__ConnectionString`) — never commit a real one. The database must already exist; `AutoCreateSchema` creates the *schema and tables*, not the database. | | `SqlServer:Schema` | string | `hist` | Must match `^[A-Za-z_][A-Za-z0-9_]{0,127}$` (it is spliced into DDL/DML) | SQL schema that holds `SchemaVersion`, `Tag`, `Sample`, `Event` and the `SampleRow`/`TimeRow` table types. | | `SqlServer:CommandTimeout` | TimeSpan | `00:00:30` | Must be `> 0` | Per-command timeout, rounded up to whole seconds (minimum 1 s). The SQL backend does **not** use `CallTimeout`. | | `SqlServer:AutoCreateSchema` | bool | `true` | — | `true`: on first use apply any missing embedded script under a `sp_getapplock` (`OtOpcUaHistorianSchema`), so concurrent nodes serialise. `false`: only verify `SchemaVersion >= 1`; if behind, **every** historian call fails with a clear error naming the missing version (surfaced in the read client's health `LastError`, never a host crash). A failed bootstrap is retried on the next call, not latched. | **Validation rules** (`ServerHistorianOptionsValidator`, driver-role nodes only; admin-only nodes never build historian singletons): - `Backend` must name a registered backend — checked **always**, with no I/O, even when nothing is enabled. - Everything else is **consumer-gated**: it runs only when `ServerHistorian:Enabled` **or** `AlarmHistorian:Enabled` is true (the alarm sink sources its connection from this section even when `ServerHistorian` is off). - `Backend=Gateway`: `Endpoint` must be an absolute http(s) URI, `ApiKey` non-empty, and `UseTls` must agree with the endpoint scheme. - `Backend=SqlServer`: the three `SqlServer` rules above. `Endpoint`/`ApiKey`/`UseTls` are **not** required. - ⚠️ `Backend=SqlServer` with a malformed `Schema` throws `ArgumentException` while the provider is built, so it stops host start even when no consumer is enabled (the validator is consumer-gated, provider construction is not). - The alarm-only warning at startup (`AlarmHistorian` on, `ServerHistorian` off) names the selected backend. > **Do not commit `ApiKey` to `appsettings.json`.** Set it via the environment variable > `ServerHistorian__ApiKey`, a secrets store, or a deployment-time overlay. The checked-in default is > always empty. > **Gateway-side prerequisites.** The target gateway must run `RuntimeDb:Enabled=true` (continuous > `WriteLiveValues`) + `RuntimeDb:EventReadsEnabled=true` (alarm-history `ReadEvents`), and the API key > must carry the scopes `historian:read`, `historian:write`, `historian:tags:write`. > **Migration from the Wonderware backend.** Rename the old keys: `Host`/`Port` → `Endpoint` > (`https://host:5222`); `SharedSecret` → `ApiKey` (env `ServerHistorian__ApiKey`); > `ServerCertThumbprint` → `CaCertificatePath` (+ `UseTls` / `AllowUntrustedServerCertificate`). The `ServerHistorian` section is independent of the `AlarmHistorian` section (the alarm write path) and the `ContinuousHistorization` section (driver-value capture). All three target the **same** gateway — but only `ServerHistorian` carries the connection (endpoint/key/TLS); the other two source it from there. > **Continuous-historization value quality on the Gateway backend is still server-stamped.** The recorder now > carries each sample's **real** OPC UA status (it used to hard-code Good `192`, so a Bad sample was historized as > Good). `GatewayHistorianValueWriter` maps that status to an OPC-DA quality byte with > `GatewayQualityMapper.ToDaQuality` (the inverse of the read-side table; unknown codes fall back by severity: > Good ⇒ 192, Uncertain ⇒ 64, Bad ⇒ 0). But the gateway's `WriteLiveValues` SQL path **discards** the quality > it is sent (a server-managed column; 0.2.0 Contracts C-002), so **AVEVA still stores every recorded value as > Good** — a pre-existing limitation of that backend, now documented rather than fixed (archreview 06/C-7). The > `SqlServer` backend stores the real status. > ✅ **Continuous value capture is LIVE-PROVEN END-TO-END > ([#491](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/491) CLOSED — gate run 2026-08-01, > completed 2026-08-04; see `docs/plans/2026-08-01-continuous-historization-live-gate.md`).** Against a > real `ZB.MOM.WW.HistorianGateway` talking to the real AVEVA Historian on `wonder-sql-vd03`: deploy → > `EnsureTags` provisioning → dependency-mux value delta → FasterLog outbox append → drain → > `WriteLiveValues` → **value read back from the historian**, plus **restart convergence** (the recorder > re-registers the deployed ref set after a process restart, with no new deployment). > > The read-back used two registers carrying **distinct non-zero** constants (`8224`, `120`), each of > which read back as its own value — a `0`-valued source cannot distinguish a correct value from a > wrong one. > > ⚠️ The gate found a **production-fatal defect**: the recorder threw `NotSupportedException` on the > **first value it ever appended**, because it touched `Self` after an `await …ConfigureAwait(false)`, > which discards Akka's ActorContext. Supervision restarted the actor, and since the recorder is seeded > with an *empty* ref set, the restart discarded every historized ref — historization stopped after one > value. Fixed; pinned by a regression test whose outbox fake completes on a **pool** thread (a fake > returning `ValueTask.CompletedTask`, or even `await Task.Yield()`, cannot reproduce it). > > The gate also found the two defects below, both since fixed. > > **Related, separately open:** `EnsureTags` provisioning fires only for **newly-added** historized > tags (`AddressSpaceApplier` iterates the added set), so flipping an **existing** tag to > `"isHistorized": true` never provisions it — see > `docs/plans/2026-06-27-historian-gateway-integration-issues.md` Issue 5. --- ## ContinuousHistorization configuration `ContinuousHistorizationOptions` binds the `ContinuousHistorization` section. It carries **only** the recorder + outbox knobs — the backend connection (gateway endpoint / key / TLS, or the SQL Server connection string) is sourced from `ServerHistorian` (above), never restated here. The outbox is a node-local FasterLog and is registered the same way for every backend. | Key | Type | Default | Description | |---|---|---|---| | `Enabled` | bool | `false` | `true` **and** a configured `ServerHistorian` wires the FasterLog outbox + the selected backend's value writer and spawns the `ContinuousHistorizationRecorder`. `false` → no recorder; driver tag values are not historized. | | `OutboxPath` | string | `""` | **Directory** holding the FasterLog segment + commit files. Required when `Enabled`. In production set an **absolute** path on durable storage — a relative path resolves against the host's working directory. | | `CommitMode` | string | `PerEntry` | `PerEntry` fsyncs before the append returns (no loss window); `Periodic` batches commits onto a background timer every `CommitIntervalMs`. Parsed case-insensitively; an unrecognized value falls back to `PerEntry`. **See the batch-commit note below — `PerEntry` is one fsync per ingest *burst*, not per value.** | | `CommitIntervalMs` | int | `100` | Periodic-mode commit cadence (ms); must be `> 0` under `Periodic`, ignored under `PerEntry`. | | `DrainBatchSize` | int | `1024` | Outbox entries peeked + written per drain pass. **Raised from 64**: the peek is an in-memory scan of the outbox's live index (widening it costs allocation, not I/O), and a pass is now bounded-parallel over `DrainConcurrency` rather than sequential, so a wide batch no longer costs `tags × latency`. At 64 a drain drew down at most 64 entries every `DrainIntervalSeconds` — below the arrival rate of even a modest historized tag set — so any gateway hiccup left the outbox permanently behind, growing toward `Capacity` and shedding oldest samples instead of catching up. | | `DrainIntervalSeconds` | double | `2` | Steady drain cadence (and the post-success reschedule). | | `DrainConcurrency` | int | `8` | Historian tags a single drain pass writes **concurrently**. One peeked batch is grouped per historian tag and each group is an independent gateway round-trip, so a sequential pass cost `tags × latency`; this window makes it `⌈tags / concurrency⌉ × latency` while bounding how many round-trips can be outstanding against the gateway at once. `1` restores the fully sequential drain; a value `< 1` is clamped to 1 (with a `Validate()` warning). **The ack is unchanged** — still all-or-nothing after every group has reported, and the #538 per-tag quarantine still keys on "another group wrote this pass". | | `Capacity` | int | `500000` | Max un-acked outbox entries before the drop-oldest policy sheds the front of the queue; `0` = unbounded. Roughly 50 MB of FasterLog. The #538 **disk backstop** for the one case the per-tag quarantine deliberately cannot resolve — a historian unreachable or refusing *every* write, where nothing proves any particular tag is at fault so the recorder correctly retains. | | `MaxWriteAttempts` | int | `5` | #538 poison-tag budget: consecutive failed drain passes one historian tag may accumulate before its queued entries are abandoned and the FIFO head is allowed to advance. **Counted only across passes where *other* tags wrote successfully**, so an ordinary gateway outage never advances it. Clamped to at least 1; each abandonment logs a Warning naming the tag. | | `MinBackoffSeconds` | double | `1` | Initial retry backoff after a failed drain pass. | | `MaxBackoffSeconds` | double | `30` | Cap on the exponential retry backoff. | > **`PerEntry` now means one fsync per ingest *burst*, not per value.** The recorder appends a whole > value burst through `IHistorizationOutbox.AppendBatchAsync`, which enqueues every entry and then > issues a **single** group commit — so a burst of N values costs one fsync rather than N. The > durability contract is unchanged: the call does not return until that commit completes, so nothing > the caller has been told is buffered can be lost to an unclean restart. (The documented residual is > unchanged too: a throw *before* the commit leaves the enqueued prefix tracked-but-uncommitted — > served correctly for the life of the process, absent after an unclean restart — exactly the > enqueue-then-commit window the single-entry path always had, widened from one entry to the batch.) > **Cross-repo follow-up — a multi-tag batch `WriteLiveValues` on the gateway contract.** The drain's > unit of work is one gateway round-trip **per historian tag**, because > `ZB.MOM.WW.HistorianGateway.Client` **0.3.0** exposes no RPC that writes values for more than one tag > in a call. Bounded parallelism (`DrainConcurrency`) is therefore the ceiling: it overlaps the > round-trips but cannot remove them, so a wide tag set still pays `⌈tags / concurrency⌉` network > hops per drain pass. Collapsing that to a single hop needs a **batch/multi-tag `WriteLiveValues`** on > the `historian_gateway.v1` contract in the `~/Desktop/HistorianGateway` sister repo. Until that > lands, raising `DrainBatchSize` widens each hop's payload but not the hop count — which is why the > concurrency window, not the batch width, is what governs drain throughput today. ### Typed values, real status and outage markers The recorder no longer assumes "a numeric Good sample". Each mux delta is coerced to a typed `HistorianValue` (`Double` / `Int64` / `Boolean` / `String` / `DateTime` / `Empty`; a `ulong` above `long.MaxValue` becomes `Double`; arrays and anything else are dropped and counted) and appended with the **real** status the driver published. A value whose kind is not in the selected backend's `SupportedKinds` is dropped at capture and counted in `RecorderStatus.DroppedUnsupportedKind` (the field was `DroppedNonNumeric` before the rename; `Empty` is supported by `SqlServer`, not by `Gateway`). ⚠️ **Comms loss is historized — as an outage marker (design §9.10).** `DriverHostActor.FanCommsLossToDependencyMux` (#481) fans `(null, BadNoCommunication)` for every value ref of a comms-lost driver to the dependency mux, and the recorder cannot tell that from a driver's own Bad publish. So: - **`SqlServer`:** each disconnect stores **one `Empty` / `BadNoCommunication` (`0x80310000`) sample per historized tag**. A driver's own Bad publish is stored the same way (`Empty` kind plus its own code — e.g. Modbus `BadCommunicationError` `0x80050000`). This is deliberate: raw reads show the gap, and aggregates and AtTime interpolation do not bridge it. Cost: one extra row per tag per outage. - **`Gateway`:** the marker is `Empty`, which that backend does not support, so it is dropped and counted in `DroppedUnsupportedKind`. ### Outbox format v2 and rollback Outbox entries are written in format **v2**: `"HV2\0"` magic + version byte `2` + id + tag + value kind + a kind-specific payload + the 32-bit status + timestamp. Entries left in an outbox by an older build (the fixed 38-bytes-plus-tag layout, always quality 192) are still **read** — as a `Double` with status `Good`. A record that matches neither layout is **skipped and counted, never thrown**, so a corrupt record cannot jam the drain (this also holds during the FasterLog recovery scan at start-up); the lifetime count is the gauge **`otopcua.historian.outbox.unreadable`** (`IHistorizationOutbox.UnreadableCount`). A growing value means undecodable records are being skipped. > **Metric prefixes differ.** The outbox's other two gauges are `otopcua.historization.outbox.depth` and > `otopcua.historization.outbox.dropped`; the unreadable gauge is **`otopcua.historian.outbox.unreadable`** > (the name the design fixed). All three describe the same outbox — search for both prefixes. ⚠️ **Rollback: drain or delete the outbox before downgrading.** A node running a pre-v2 build cannot read v2 records — its outbox deserializes every record in its constructor and fails to start on the first v2 one. Before rolling a node back, let the outbox drain to zero (`otopcua.historization.outbox.depth`) or stop the node and delete the directory named by `ContinuousHistorization:OutboxPath` (accepting the loss of anything un-drained). Upgrading needs no action. **Delivery semantics on `SqlServer`.** Writes are *insert-if-absent* on `(TagId, SourceTimeUtc)`: the outbox's at-least-once redelivery (and both pair nodes writing the same sample in a boot window) converges on one row, and the first-stored sample wins. **Duplicate timestamps inside one batch collapse to the first.** A `NaN` / `±Infinity` `Double` cannot be stored in a SQL `float`, so the **value writer stores it as an `Empty` sample** — the tag is not quarantined for one bad reading (HistoryUpdate behaves differently: see [below](#historyupdate)). ⚠️ **An `Empty` sample is always Bad** (the quality contract: a stored sample is never served better than "no value"): a non-finite `Double` that arrived with a Good or Uncertain status is stored as `Empty` + **`BadOutOfRange`** (`0x803C0000`), and any other `Empty` with a non-Bad status as `Empty` + **`BadNoData`** (`0x809B0000`). An existing Bad code — the comms-loss `BadNoCommunication` marker, a driver's own Bad publish — is stored as it is. A duplicate-key error (2627/2601) rolls the whole statement back, so the writer re-runs the statement (at most twice) instead of treating the error as success. --- ## A poison tag no longer pins the outbox (#538) `DrainOnceAsync` peeks a batch, writes **every** per-tag group in it, and then acks **all-or-nothing** — because the outbox's `RemoveAsync` truncates the FIFO head *through* the acked id, so acking one tag's entries could silently drop older, unwritten entries belonging to another. That much is deliberate and unchanged. What was wrong is what happened when a tag failed **permanently**: the head never advanced, so every **healthy** tag in the batch was re-submitted to the historian on every drain pass, indefinitely — duplicate samples at identical timestamps, and an outbox that never truncated (observed live at 8.3 MB from a single unprovisionable tag, with the gateway's `ok` **and** `error` counters climbing together — the signature of this bug). The drain now tracks **consecutive failures per historian tag** and abandons a tag's queued entries once it exceeds `ContinuousHistorization:MaxWriteAttempts` (default 5), letting the head advance. Two properties matter: - **A pass where NOTHING wrote abandons nothing.** The counter only advances on a pass where at least one *other* tag wrote successfully — that is what distinguishes "this tag is poison" from "the backend is down". A count-only quarantine looks identical on the poison-tag case and silently discards the whole backlog during an ordinary gateway restart, turning a duplicate-samples bug into a lost-history one. Pinned by a falsification-control test (`A_backend_that_fails_every_tag_abandons_nothing`) that goes red when the evidence check is removed while the poison-tag test stays green. - **A tag over budget waits for the others.** Abandonment is deferred until *every* failing tag in the batch is over budget, so a tag is never discarded on a pass that could not have advanced the head anyway. Each abandonment logs a Warning naming the tag and the discarded sample count, and the total rides `RecorderStatus.AbandonedEntries`. The most common cause is a tag the historian has no definition for — easy to reach, because `EnsureTags` provisioning runs only for newly-**added** historized tags, so a changed `historianTagname` is never provisioned (Issue 5 above). `ContinuousHistorization:Capacity` now also defaults to **500,000** entries rather than unbounded. It is the disk backstop for the one case the quarantine deliberately cannot resolve — a historian refusing *every* write, where retaining is correct. Set `0` to restore unbounded retention. ## Only the recording Primary captures (#537) Both nodes of a redundant pair serve the same address space and see the same dependency-mux fan-out, so an ungated recorder wrote **every sample twice** (measured: the gateway's write counter advanced twice per value with both centrals up, and halved when one was stopped). The recorder now reads a capture gate per value, fed from `PrimaryGatePolicy.ShouldRecordHistorizedValues` via `IHistorizationRoleView` — the same shape as the alarm store-and-forward sink's `drainGate`, with two deliberate differences: - **It gates the CAPTURE, not the drain.** The alarm queue lives in LocalDb and *replicates*, so the Secondary already holds the Primary's rows and gating the drain loses nothing. The historization outbox is a node-local FasterLog replicating to nobody: gating only its drain would leave the standby appending forever to a queue no one empties — unbounded disk, and a flood of stale duplicates the moment it is promoted. Entries a node already captured are still drained by that node, so a demotion loses nothing recorded. - **The relevant peer is one that records the same refs**, not one that holds the same rows. A node stands down only for a Primary in its own per-cluster mesh. A node carrying no cluster-scoped role cannot establish that, so it never stands down: under a fleet-wide election the elected Primary is usually running a different cluster's configuration, and deferring to it would end this cluster's historization silently — the drain would report success on an empty outbox forever. That is the same scope mismatch that once suspended the alarm drain on every node of the docker-dev rig at once. An unknown role **captures**, matching the alarm drain: a duplicate sample is recoverable, a hole in the trend is not. The residual cost is a capture gap across a failover, bounded by how long the newly promoted node takes to see its first redundancy snapshot — inherent to any non-duplicating design. Suppressed values are counted in `RecorderStatus.SuppressedNotPrimary`, and each gate transition logs once. --- ## Tag auto-provisioning (EnsureTags) When a deploy adds a historized tag, the server auto-ensures it exists in the historian via the gateway's `EnsureTags` (Gateway) or the `Tag`-row `MERGE` (SqlServer) **before** the first value or HistoryRead lands. `AddressSpaceApplier.Apply()` fires a **non-blocking, fire-and-forget** `IHistorianProvisioning.EnsureTagsAsync` for each added historized value tag (with the resolved historian name — the `historianTagname` override when set, else the driver `FullName`); the hook is wrapped so a faulted, slow, or throwing provisioner can **never** block or fail a deploy. On the **Gateway backend**, non-numeric (`String`/`DateTime`/`Reference`) types are skipped, not provisioned; the SqlServer backend provisions every type (below). **Wiring + gating.** The concrete provisioner comes from the selected backend's provider (`CreateProvisioning`): the gateway-backed `GatewayTagProvisioner`, or — for `SqlServer` — `SqlServerTagProvisioner` (one `MERGE … WITH (HOLDLOCK)` upsert per chunk of 400 tags; every data type is provisionable so `skipped` is always 0; a blank or over-400-character name fails alone; a null unit/description in a request keeps the stored value). It is registered by the Host's `AddHistorianProvisioning` and gated on the **same** `ServerHistorian:Enabled` flag as the read path — both use the one resolved backend. When the section is disabled (or absent) the applier resolves the no-op `NullHistorianProvisioning` and **provisioning does not run** (so on a read-only-historian deployment, ensure tags exist by some other means). On the Gateway backend the API key must carry the `historian:tags:write` scope for `EnsureTags` to succeed. **Observability.** Every dispatch logs an Information tally — `historian provisioning completed (dispatched=N, requested=…, ensured=…, skipped=…, failed=…)`. A line showing `dispatched=N` but `requested=0` indicates the **no-op** provisioner ran (provisioning is not wired for this node), which is the signal to check that `ServerHistorian:Enabled=true`. > **Why this matters for HistoryRead latency.** Until a tag is provisioned, the historian does not know it, > so a HistoryRead on it runs to the full `CallTimeout` (~30 s) before erroring. Auto-provisioning on deploy > keeps freshly-historized tags from hitting that slow path. See the `CallTimeout` row above. ### Deferred types and the UInt16 fallback (#600 item 4) > **Gateway backend only.** The `SqlServer` backend provisions and records every type (`String` and `DateTime` included). Historization on the **Gateway** backend is **numeric-analog only**. `HistorianTypeMapper` classifies exactly nine types as historizable (`Boolean`, `Int16/32/64`, `UInt16/32/64`, `Float32`, `Float64`): - **Deferred: `String`, `DateTime`, `Reference`.** The historian's SQL analog write path cannot carry them. Provisioning skips them (counted in the `skipped=` tally, via `IsHistorizable`), the mapper throws `NotSupportedException` for them, and `ContinuousHistorizationRecorder` drops their values and counts them in `DroppedUnsupportedKind` (renamed from `DroppedNonNumeric`). A tag authored `isHistorized` with one of these types therefore silently records nothing; HistoryRead on it returns whatever the historian holds (typically `GoodNoData`). - **Fallback: `UInt16 -> Uint4`.** The historian's `UInt2` write path is unproven upstream, so UInt16 tags are provisioned as the wider `UInt4`. Values are unaffected; only the historian-side storage type differs. (`Boolean -> Int2` is the other fallback, below.) Lifting either needs gateway-side support first; neither is scheduled. ### Boolean historization maps to Int2 (issue #441) `EnsureTags` is contract-documented (HistorianGateway 0.2.0+) as an **upsert** — *"creates or updates historian tags according to the given definitions"* — that maps per-tag onto AVEVA's `HistorianClient.EnsureTagAsync(definition) -> bool`. It is **not idempotent** (not routed through the idempotent retry pipeline) and **not skip-if-exists**. `HistorianTypeMapper` maps **`DriverDataType.Boolean → HistorianDataType.Int2`** and Boolean values historize as a small **0/1 integer**. Int2 is used rather than the semantically-exact `Int1` because the historian's **`Int1` analog-tag creation path is server-degenerate** — `EnsureTags(Int1)` stores an unusable stub, so a Boolean tag mapped to `Int1` never becomes historizable. This is an evidence-based decision from a live probe against the deployed 2023 R2 historian; the root cause and gateway-side resolution are tracked in [issue #441](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/441) and HistorianGateway [#11](https://gitea.dohertylan.com/dohertj2/historiangw/issues/11) / PR #16 (which documents `Int1` as an explicit, evidence-based exclusion). The value-write path is unaffected: the recorder sends every value as a `double` on `WriteLiveValues` regardless of tag type, so a Boolean `0.0/1.0` round-trips cleanly into the Int2 tag. **Observed live (2026-07-13, real 0.2.0 gateway → real AVEVA historian on `wonder-sql-vd03`, probed via `grpcurl` — archreview R2-06 T12):** - `EnsureTags` is a **real create-or-update upsert** — re-`EnsureTags` of an existing tag at the same type returns `Success=true`. - An **in-place retype between SUPPORTED types is ACCEPTED** — `Float→Double` on an existing tag returns `Success=true`. - A type sweep confirmed **`Int2/UInt2/Int4/UInt4/Float/Double` all provision `Success=true`; only `Int1` fails** (`Success=false, Error="ProtocolEvidenceMissingException"` — the vendored client's "no capture evidence for this type" guard → gRPC `Unimplemented`), both for a fresh tag and as a retype target. Mapping Boolean to `Int2` sidesteps that gap entirely. > **Prerequisite for provisioning to land at all:** the provisioner must stamp a **positive `StorageRateMs`** > — a `0` made `EnsureTags` throw `ArgumentOutOfRangeException` and fail *every* provision (fixed in > `GatewayTagProvisioner`, PR #439). **Migration note for pre-existing Boolean tags.** A Boolean tag historian-side as Float (e.g. from the 0.1.0-era default) is asked, on its next deploy, to upsert to Int2. Both are supported analog types, so the retype is accepted. Provisioning is fire-and-forget and never blocks/fails a deploy; a per-tag rejection (if any) surfaces only as the tally's `failed` count, and values written to a still-Float tag remain readable (Float supersets the 0/1 range) pending reconcile. A bulk retype sweep is **not** done in code. --- ## HistoryRead behavior ### Read variants The server supports all four OPC UA HistoryRead variants: | Variant | Node type | CLI `--aggregate` | |---|---|---| | **Raw** | Historized variable | (omit `--aggregate`) | | **Processed** | Historized variable | `Average`, `Minimum`, `Maximum`, `Count`, `Start`, `End`, `StandardDeviation` (the CLI's own list; `SqlServer` serves thirteen aggregates — see the [aggregate table](#aggregates-processed-reads)) | | **AtTime** | Historized variable | n/a (client supplies exact timestamps) | | **Events** | Equipment folder (event notifier) | n/a | **Variable nodes** (historized tags) serve Raw, Processed, and AtTime history. `Historizing=true` and `AccessLevels.HistoryRead` are set at materialization so any compliant OPC UA client can discover historized capability from the node's attributes. > **v3 Batch 4 — dual-namespace registration.** A historized raw tag surfaces as **two** > variable nodes: the Raw-namespace node (`ns=Raw, s=`) and, for each referencing > equipment, a UNS-namespace node (`ns=UNS, s=///`). > **Both NodeIds register the SAME historian tagname** (the raw tag's `FullName` / > `historianTagname`), so HistoryRead against either NodeId resolves to the same tagname and > returns the same series. The historization intent is single-sourced — the mux's > `HistorizedTagRef` set stays keyed by RawPath (no doubles), and continuous historization / > `EnsureTags` provisioning run once per raw tag regardless of how many equipment reference it. **Equipment-folder event-notifier nodes** serve Event history. Every equipment folder that owns at least one alarm condition is already an event notifier; the server registers a `sourceName` (the equipment id) for each such folder and maps event history reads to the HistorianGateway using that source. (Alarm-history `ReadEvents` requires the gateway running `RuntimeDb:EventReadsEnabled=true`.) Event-field projection supports the standard `BaseEventType` select clauses — `EventId`, `SourceName`, `Time`, `ReceiveTime`, `Message`, and `Severity`; an unsupported select operand returns a null field (spec-conformant). ### No driver-side history (Gitea #655) History is served **only** by the server-wide `IHistorianDataSource` (`GatewayHistorianDataSource` / `NullHistorianDataSource`). No driver contributes history: the OpcUaClient driver does **not** proxy its upstream server's HistoryRead (the former `IHistoryProvider` passthrough was never dispatched to and was deleted). On an OpcUaClient tag, `isHistorized` means "OtOpcUa records this value into the historian from now on", not "forward history reads upstream". A node without `isHistorized` has no HistoryRead access bit and answers `BadHistoryOperationUnsupported`. ### Graceful degradation | Situation | HistoryRead status | |---|---| | Historized node, historian configured and reachable, results found | `Good` | | Historized node, historian configured, time range empty | `GoodNoData` | | Historized node, historian NOT configured (`Enabled=false` / Null source) | `GoodNoData` (empty) | | Non-historized node | `BadHistoryOperationUnsupported` | | Backend timeout or exception | `BadHistoryOperationUnsupported` per node; other nodes in the same batch are unaffected | | `Backend=SqlServer`, `AutoCreateSchema=false`, schema missing/behind | every read/write fails with a clear error naming the missing schema version (health `LastError`); reads answer `BadHistoryOperationUnsupported` per node; no host crash | A historized node with no historian configured never returns an error status — it returns empty. This means a deployment can author and publish historized tags before the historian gateway is provisioned, without producing error spikes in connected clients. ### Continuation-point paging (Raw) `HistoryRead-Raw` is paged server-side. The historian backend is single-shot (it returns up to `NumValuesPerNode` samples with no continuation point of its own), so the server synthesises paging time-based: - A page that returns **exactly** `NumValuesPerNode` samples (with `NumValuesPerNode > 0`) MAY have more behind it, so the server stores a resume cursor and returns an opaque continuation point (16 bytes). The client hands it back to fetch the next page. - A short page (fewer than the cap) is the last page — no continuation point. - `NumValuesPerNode == 0` ("all values, no limit") is never paged; the whole window returns in one shot. - The resume cursor is **tie-safe**: the next page resumes from the last returned sample's SourceTimestamp *inclusive* and drops the boundary samples already emitted, so samples sharing the boundary timestamp are neither duplicated nor skipped. > **Oversized tie clusters — within-timestamp paging.** When more samples share one > `SourceTimestamp` than the current page cap, the server detects that the cursor has stalled on > a tie cluster (the last returned timestamp equals the resume timestamp). It then **over-fetches > the entire cluster** at that single timestamp up to a bounded ceiling controlled by > `ServerHistorian:MaxTieClusterOverfetch` (default **65 536**), then serves the cluster > `NumValuesPerNode` samples at a time across successive pages, advancing the cursor one tick past > the timestamp once the cluster is fully drained. > > **Reverse reads** (`start > end`, newest first) page the same way in the opposite direction: a drained > cluster steps the cursor one tick **down**, the read ends when that would reach the (excluded) `end`, and > the over-fetched cluster is sliced in descending order. Before the fix a reverse read with a small > `NumValuesPerNode` stopped after its first page with `GoodNoData`. > > **Short pages within a cluster still carry a continuation point.** A within-cluster page that > returns fewer than `NumValuesPerNode` samples (because the cluster happened to be smaller than > the cap, or is the final partial batch) is not the last page if the cluster itself has not been > fully emitted — the server retains the continuation point so the client can drain the remainder. > Only when the cluster is exhausted and the cursor has advanced past the timestamp does the > short-page rule apply. > > **Cluster larger than `MaxTieClusterOverfetch`.** If the over-fetch itself reaches the ceiling > without spanning the full cluster, the node fails **loudly** with > `BadHistoryOperationUnsupported` and the tag + timestamp + ceiling are logged. Remedies: raise > `MaxTieClusterOverfetch` (or `NumValuesPerNode`) to cover the full cluster, or investigate the > data anomaly (raw samples normally carry strictly increasing distinct timestamps). > > For a single tag's raw history a tie cluster larger than the default 65 536 is a severe data > anomaly. The ceiling exists to bound server-side memory on pathological data, not to cap normal > operation. Continuation points are bound to the OPC UA session (the SDK's `ServerConfiguration.MaxHistoryContinuationPoints` cap, default 100, with oldest-eviction; points are disposed when the session closes). Resuming an unknown / evicted / released point returns `BadContinuationPointInvalid`. `releaseContinuationPoints` drops the stored cursors without reading data. ### Total aggregate The OPC UA `Total` aggregate is **supported** over the HistorianGateway backend. The gateway exposes a native **`Integral`** retrieval mode, so `Total` maps straight to it (`HistoryAggregateType.Total → RetrievalMode.Integral`) — no client-side scaling. (This replaces the retired Wonderware path, which had no `Total` column and derived it client-side as time-weighted `Average × interval-seconds`.) `Count` is likewise a native gateway mode. Bucket status codes and timestamps are preserved unchanged; empty / null buckets surface as `BadNoData`. ### Aggregate set and `BadAggregateNotSupported` (#285) `HistoryAggregateType` carries the wider Part 13 set (Interpolative, TimeAverage, Start, End, Delta, Range, Min/MaxActualTime, Start/EndBound, DeltaBounds, DurationGood/Bad, PercentGood/Bad, WorstQuality, NumberOfTransitions, AnnotationCount, StandardDeviationPopulation, VariancePopulation); `OtOpcUaNodeManager.MapAggregate` maps each standard aggregate NodeId. The **`SqlServer`** backend serves the thirteen listed in the [aggregate table](#aggregates-processed-reads); everything else (e.g. `Interpolative`, `TimeAverage`, `StandardDeviationPopulation`) is `BadAggregateNotSupported` for that node. The **HistorianGateway** backend only has retrieval modes for the original five (Average, Minimum, Maximum, Total, Count): for any other member `AggregateModeMapper` throws `HistoryAggregateNotSupportedException`, which the HistoryRead path turns into `BadAggregateNotSupported` for that node only (not a generic failure, and not counted as a backend fault). The OPC UA Client driver forwards every member upstream. A NodeId outside the mapped set (e.g. `StandardDeviationSample`) is still `BadAggregateNotSupported`. ### Known limitations - **Processed and AtTime are single-shot** (no continuation points). Unlike Raw, neither `ReadProcessedDetails` nor `ReadAtTimeDetails` carries a client count cap (`NumValuesPerNode`): the Processed bucket count is deterministic (window / interval) and AtTime returns exactly one sample per requested timestamp, so the single-shot backend returns the complete result in one read and there is no "full page ⇒ maybe more" signal to page on. Returning the full result with no continuation point is spec-conformant. - **No modified-value history** (`HistoryReadModified`). Requests for modified values return `BadHistoryOperationUnsupported`. This is **infra-gated, not a server-code gap**: the HistorianGateway backend (`GatewayHistorianDataSource`) exposes only a current-value read path — there is no modified/edited-history surface to source the data from. The server-side override is in place (it cleanly rejects modified reads per node) and `IsReadModified` is honoured; serving real modified-value history is unblocked only once the gateway grows a modified-read RPC. Until then, rejecting is the correct, spec-conformant behaviour. ### Redundancy and authorization History reads are served from any node — there is no Primary gate. The `HistoryRead` AccessLevel bit set at materialization admits a node to the read dispatch. With `Security:Acl:Enabled=false` (the default) HistoryRead is **ungated** for any admitted session; with it `true`, the scoped `HistoryRead` grant decides each node (see [security.md](security.md#data-plane-authorization)). **HistoryUpdate is always gated** — see [HistoryUpdate](#historyupdate). --- --- ## SQL Server backend Selected with `ServerHistorian:Backend=SqlServer`. Implemented in `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer` (the project does not reference the Gateway project). All five seams share one connection helper, one schema bootstrapper and one tag catalog, so the schema is ensured once per process and tag ids are cached once. ### Schema Created by the embedded idempotent script `Schema/001_initial.sql` (mirrored at [`docs/historian/sqlserver/001_initial.sql`](historian/sqlserver/001_initial.sql)); `$(Schema)` is the configured schema. | Object | Purpose | |---|---| | `SchemaVersion(Version, AppliedUtc)` | Applied-script ledger. Required version: **1**. | | `Tag(TagId int identity, Name nvarchar(400) unique, DataType tinyint, EngineeringUnit, Description, CreatedUtc, UpdatedUtc)` | One row per historian tagname. `DataType` is a `DriverDataType`. | | `Sample(TagId, SourceTimeUtc datetime2(7), ValueKind tinyint, NumValue float, IntValue bigint, StrValue nvarchar(max), StatusCode int, ServerTimeUtc)` | **PK `(TagId, SourceTimeUtc)`** — one value per timestamp (Part 11), which makes redelivery idempotent and rules out tie clusters. `StatusCode` is the uint status stored bit-for-bit. | | `Event(EventId nvarchar(64) PK nonclustered, SourceName, AlarmId, AlarmName, AlarmTypeName, Severity, EventKind, Message, UserName, Comment, EventTimeUtc, ReceivedTimeUtc)` | Alarm history. Clustered on `(EventTimeUtc, EventId)`; extra index `(SourceName, EventTimeUtc)`. | | `SampleRow`, `TimeRow` table types | TVPs for batched sample writes/merges, AtTime and delete-at-time. | Column mapping per `ValueKind`: `Double` → `NumValue`; `Int64` → `IntValue` + `NumValue` (as a double, for aggregates); `Boolean` → `IntValue` 0/1 + `NumValue` 0/1; `String` → `StrValue`; `DateTime` → `IntValue` (UTC ticks); `Empty` → all NULL. `ValueKind` codes: Empty 0, Double 1, Int64 2, Boolean 3, String 4, DateTime 5. Timestamps are bound as `datetime2(7)` everywhere (full 100 ns precision — never a `DataTable`, which would round to 3.33 ms). **Applying the script by hand** (needed when `AutoCreateSchema=false`): create the database, then `sqlcmd -S -U -P -d -I -v Schema=hist -i docs/historian/sqlserver/001_initial.sql` (`-v Schema=` supplies the `$(Schema)` placeholder; verified against the docker-dev SQL container). Re-running is a no-op. ### Writes - **Live values** (`IHistorianValueWriter`): resolve `TagId` through a cache, creating the `Tag` row on a miss (`MERGE … WITH (HOLDLOCK)`; `DataType` from the first non-Empty value's kind, provisioning corrects it later), then `INSERT … SELECT … WHERE NOT EXISTS` through the `SampleRow` TVP. A null timestamp is stamped now; `ServerTimeUtc = SYSUTCDATETIME()`. It returns `false` (never throws) on a SQL failure so the recorder's retry/quarantine owns it. Semantics: [delivery semantics](#outbox-format-v2-and-rollback). - **Alarm events** (`IAlarmHistorianWriter`): per event, insert-if-absent keyed by a deterministic `EventId` = lowercase-hex SHA-256 over the length-prefixed fields `EquipmentPath, AlarmId, EventKind, TimestampUtc.Ticks, User, Comment, Message`. Outcomes: `Ack` on insert or already-present (including a concurrent writer winning a 2627/2601 race); `RetryPlease` on transient SQL errors (connection, timeout, deadlock 1205 …), bootstrap failures, cancellation and any unclassified error; `PermanentFail` (dead-letter) only for data errors SQL will reject forever (truncation 2628/8152, conversion, null, range, constraint). `SourceName` is the event's `EquipmentPath`; `Severity` is the numeric Part 9 value (Low 200, Medium 500, High 700, Critical 900). - **Tag names** are `nvarchar(400)`, and a longer name is **refused, never truncated** (a truncating parameter would resolve it onto whichever tag owns its first 400 characters): `EnsureTags` fails it individually, the live-value writer returns `false` for that tag (a per-tag failure, so the #538 quarantine eventually abandons it — logged as a Warning), the updater answers `BadInvalidArgument` for every item, and reads treat it as not found (Raw empty, AtTime / Processed `BadNoData`). An event read filtered by a source name over 400 characters returns no events. Keep historian tagnames under 400 characters. - **Schema bootstrap failures are cached for 5 s.** With `AutoCreateSchema=false` and the schema missing (or SQL down), every historian call fails fast with the last bootstrap error for 5 s before SQL is tried again, so a missing schema does not serialise every write, read and alarm event behind one round-trip each; each retry that fails logs a Warning. If `sp_releaseapplock` fails after an auto-create, the connection pool is cleared so the session holding the applock is closed rather than returned to the pool. ### Raw reads `ReadRawAsync` returns every sample as stored — outage markers (`Empty` + Bad status) included. The one exception is defensive: a row with no value whose stored status is **not** Bad (only a build from before the writer forced `Empty` samples Bad could have written one) is served with `BadNoData`, on Raw and AtTime alike. | Request | Window | Order | |---|---|---| | `start < end` | `[start, end)` | ascending | | `end < start` (Part 11 reverse) | `(end, start]` | descending | | `start == end` | the sample exactly at that time | — | `NumValuesPerNode == 0` is unbounded (larger values clamp to `int.MaxValue`). An unknown tag is an empty result (`GoodNoData`); reads **never create** tag rows. The node manager's paging is unchanged; because the PK rules out tie clusters, resuming "from the last timestamp inclusive and trimming it" is exact. ### AtTime reads One statement per call: the distinct requested times go in a `TimeRow` TVP and each picks up its previous-or-equal sample `p` and next sample `n` via two `OUTER APPLY`s. A *usable* sample has severity Good or Uncertain **and** a value (an `Empty` sample is an outage marker and counts as Bad whatever its code). Per requested time `T` (the result is stamped `T`): | Case | Result | |---|---| | `p` at exactly `T` | the stored value and status, unchanged | | no `p` (before the first sample), or unknown tag | `BadNoData`, null | | **`p` is Bad** (`T` lies inside an outage) | **null value carrying `p`'s Bad status code** (info bits stripped) — never better than its source | | `p` usable, **`n` Bad** | **`p`'s value held** (not interpolated into the outage), `UncertainDataSubNormal` + Interpolated | | `Double` / `Int64` with a usable `n` | linear interpolation of `NumValue`; `Int64` is computed **through `double`** and rounded to the nearest integer (ties away from zero), keeping the `Int64` type. Status `Good` + Interpolated when both neighbours are Good, else `UncertainDataSubNormal` + Interpolated | | `Boolean` / `String` / `DateTime`, **or `T` after the last sample** | **stepped**: `p`'s value with `p`'s status + Interpolated | | `T` after the last sample **and after now** | `p`'s value held, `UncertainDataSubNormal` + Interpolated (the value is not known to still hold) | Setting Interpolated (or, in aggregates, Calculated) first clears any historian bit the stored status already carries, so a result never has both. ⚠️ Because `Int64` interpolation goes through `double`, magnitudes beyond 2^53 lose precision. ### Events `[start, end)` by `EventTimeUtc`; **a reversed window (`end < start`) is swapped and returned ascending**. A null `sourceName` means all sources, otherwise `SourceName = @source`. Ordered `(EventTimeUtc, EventId)` ascending. A bounded read fetches `maxEvents + 1` rows and returns a non-null continuation point when the extra row exists; the node manager does not currently issue or resume event continuation points, so a client-capped read returns only the first page. ### Aggregates (Processed reads) > **These are simple per-sample definitions, not full OPC UA Part 13.** There is no time-weighting, no interpolated interval-boundary > value and no partial-interval status. They are pushed down to T-SQL and suit trend-style consumption. Any `HistoryAggregateType` not > in the table throws `HistoryAggregateNotSupportedException` ⇒ `BadAggregateNotSupported` for that node (e.g. `Interpolative`, > `TimeAverage`, `StartBound`, `DurationGood`, `StandardDeviationPopulation`). **Bucketing.** One result per interval of `[start, end)`; bucket *i* = `[start + i·interval, start + (i+1)·interval)`, start-inclusive / end-exclusive to the tick, the last bucket truncated at `end`. `interval <= 0` ⇒ a single interval `[start, end)`; `start == end` ⇒ no intervals. The bucket index is computed in **100 ns ticks** (not milliseconds): `DATEDIFF_BIG(microsecond, @start, t) × 10` plus a sub-microsecond correction from the two tick digits, divided by the interval in ticks. (`DATEDIFF` counts unit *boundaries crossed*, so a millisecond formulation misplaces samples near a bucket edge, and a nanosecond one overflows past ≈292 years.) A window of more than **100 000 intervals** is refused before any I/O and reported `BadHistoryOperationUnsupported` for that node. **Reverse windows** (`end < start`) are swapped, computed forward **anchored at the earlier time**, and returned in **descending** bucket order. Every returned status carries the historian *Calculated* info bit, `BadNoData` included. **Sample classes.** *Usable* = severity Good or Uncertain **and** a stored value (not `Empty`). *Good* = usable with severity Good. *Bad* = not usable. | Aggregate | Computed over | Value timestamp | |---|---|---| | `Average`, `Minimum`, `Maximum`, `Total` | `AVG` / `MIN` / `MAX` / `SUM` of `NumValue` over usable samples | bucket start | | `Count` | number of usable samples (`Int32`) | bucket start | | `Range` | `Maximum − Minimum` | bucket start | | `Start`, `End` | first / last usable sample by time (keeps the sample's type) | bucket start | | `Delta` | `End − Start` (numeric samples) | bucket start | | `MinimumActualTime`, `MaximumActualTime` | the usable sample with the smallest / largest `NumValue`; ties ⇒ earliest | **the sample's own time** | | `PercentGood` | `100 × (Good samples) / (all samples)` | bucket start | | `PercentBad` | `100 × (Bad samples) / (all samples)` | bucket start | Status and edge rules: - An **empty interval** (no samples) ⇒ `BadNoData`. **`Count` — and every value aggregate — over an interval whose samples are all Bad ⇒ `BadNoData`**, not `0`. - Value aggregates are `Good` when every sample in the bucket is Good and `UncertainDataSubNormal` when the bucket also holds any Bad **or Uncertain** sample. - `Count`, `Start`, `End`, `PercentGood`, `PercentBad` accept any value kind; the other aggregates (and `Delta`) on a bucket that holds a usable `String`/`DateTime` sample ⇒ `BadAggregateInvalidInputs` for that interval. - **`PercentGood` / `PercentBad` and Uncertain samples:** Uncertain samples are counted in the denominator but are **neither Good nor Bad**, so `PercentGood + PercentBad < 100` whenever a bucket holds Uncertain samples. Both report status `Good` for any non-empty bucket. - Everything except `Count` and `Start`/`End` is a `double`; `Int64` samples aggregate as doubles and `Boolean` as 0/1. ### Health The read client reports `GetHealthSnapshot()` (query/failure counters, consecutive failures, `LastError` — e.g. a missing schema under `AutoCreateSchema=false`). A SQL error on a read surfaces as `BadHistoryOperationUnsupported` for **that node only**; a cancelled read (deadline, client abort) is not counted as a backend fault. ### Docker-dev recipe The docker-dev `sql` container (`otopcua-dev-sql-1`, host port `14330`, `sa` with the committed dev password from `docker-dev/docker-compose.yml`) already hosts the ConfigDb; give the historian its own database on the same instance. ```bash # 1. Create the database (the backend creates the schema + tables on first use, not the database). sqlcmd -S localhost,14330 -U sa -P 'OtOpcUa!Dev123' -C -I -Q "CREATE DATABASE OtOpcUaHistorian" # 2. Point the central nodes at it. Add to the `environment:` of central-1 AND central-2 in docker-dev/docker-compose.yml # (or a compose override), rebuild the images (the rig image goes stale), then `up -d`: # ServerHistorian__Enabled: "true" # ServerHistorian__Backend: "SqlServer" # ServerHistorian__SqlServer__ConnectionString: "Server=sql,1433;Database=OtOpcUaHistorian;User Id=sa;Password=OtOpcUa!Dev123;TrustServerCertificate=True" # ContinuousHistorization__Enabled: "true" # ContinuousHistorization__OutboxPath: "/app/data/historization-outbox" # (ServerHistorian__Endpoint / __ApiKey are not used by this backend.) # 3. Mark a rig tag historized ("isHistorized": true in its TagConfig), deploy, then watch rows land: sqlcmd -S localhost,14330 -U sa -P 'OtOpcUa!Dev123' -C -I -d OtOpcUaHistorian \ -Q "SELECT t.Name, COUNT(*) n, MAX(s.SourceTimeUtc) last FROM hist.Sample s JOIN hist.Tag t ON t.TagId=s.TagId GROUP BY t.Name" ``` Only the Primary of a pair records, so the row count advances once per published value, not twice. Read it back with `Client.CLI historyread` and rewrite it with `Client.CLI historyupdate` ([Client.CLI.md](Client.CLI.md)). The password above is the committed docker-dev credential, not a secret; use `${secret:…}` / environment for anything real. ### Integration tests `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests` (bootstrap incl. two concurrent bootstrappers, every kind round-trip, redelivery, raw / AtTime / aggregate / event reads, every update result code) is gated on the environment variable **`HISTORIAN_SQL_TEST_CONNECTION`** — a connection string with `CREATE`/`DROP DATABASE` rights. Absent ⇒ every test skips. Present ⇒ each run creates a throwaway `OtOpcUaHistorian_T_` database and drops it on dispose. ```bash export HISTORIAN_SQL_TEST_CONNECTION='Server=localhost,14330;User Id=sa;Password=OtOpcUa!Dev123;TrustServerCertificate=True;Encrypt=False' dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests ``` --- ## HistoryUpdate OPC UA **HistoryUpdate** lets a client insert, replace, update and delete stored raw history. It is available only when the selected backend provides an updater — **`SqlServer` does; `Gateway` does not**, and an updater is registered only when `ServerHistorian:Enabled=true`. Updates go straight to the backend (not through the outbox), are **not Primary-gated** (the database is shared), and share the HistoryRead batch limiter and `HistoryReadDeadline`. ### Operations | OPC UA request | Server override | Per-item results (`OperationResults`) | |---|---|---| | `UpdateDataDetails`, `PerformInsertReplace = Insert` | `HistoryUpdateData` | `GoodEntryInserted`; `BadEntryExists` if a sample already exists at that timestamp | | …`Replace` | `HistoryUpdateData` | `GoodEntryReplaced`; `BadNoEntryExists` if none exists | | …`Update` | `HistoryUpdateData` | `GoodEntryInserted` or `GoodEntryReplaced` (upsert) | | `DeleteRawModifiedDetails` (`IsDeleteModified=false`) | `HistoryDeleteRawModified` | one node-level code: `Good`; `BadNoData` if `[start, end)` held nothing | | `DeleteAtTimeDetails` | `HistoryDeleteAtTime` | per time: `Good`; `BadNoEntryExists` | Each backend call is one transaction (`MERGE … OUTPUT $action` against the TVP). The node-level `StatusCode` stays `Good` when item results are mixed (Part 11 reports item outcomes in `OperationResults`); it is Bad only for whole-node failures. **Per-item rejections** (answered without touching SQL; other items still proceed): a null item ⇒ `BadInvalidArgument`; `SourceTimestamp == DateTime.MinValue` ⇒ `BadInvalidTimestamp`; a value the historian cannot type (array, unsupported CLR type) ⇒ `BadTypeMismatch`; a **`NaN` / `±Infinity` `Double` ⇒ `BadInvalidArgument`** — it cannot live in a SQL `float`, and unlike the live-value writer (which stores it as `Empty`) an operator's explicit update is rejected rather than silently rewritten; and a **null value with a Good or Uncertain status ⇒ `BadInvalidArgument`** — an `Empty` sample is always Bad, so a null value is accepted only with a Bad status (an operator-authored outage marker). A caller-supplied status code is otherwise stored as given. **No DataType check.** HistoryUpdate does **not** compare a value's type with the node's `DataType`: a value is stored as sent (a `String` written to a `Double` node's history is stored and later read back as a `String`), exactly as the live recorder stores whatever type the driver publishes. Only a value the historian cannot represent at all is refused (`BadTypeMismatch`, above). Part 11 §6.8 permits a server to answer `BadTypeMismatch` for a value of the wrong type; this server deliberately does not. **`DeleteRawModifiedDetails` needs both bounds:** a `StartTime` or `EndTime` of `DateTime.MinValue` (the wire's "unspecified") is `BadInvalidArgument` for that node and never reaches the backend. **Duplicate timestamps inside one request** (compared at 100 ns): *Insert* — the first occurrence is applied, later ones are `BadEntryExists`; *Replace* / *Update* — the last one wins and every earlier occurrence reports the winner's code; *DeleteAtTime* — the first reports the real outcome, later ones `BadNoEntryExists`. **Unknown tag:** Insert / Update create the tag row; Replace and the deletes report `BadNoEntryExists` (DeleteRaw: `BadNoData`) without creating anything. **Unsupported ⇒ `BadHistoryOperationUnsupported`** (per node): the node is not historized; the backend has no updater (`Gateway`, or `ServerHistorian:Enabled=false`); `PerformUpdateType.Remove`; `DeleteRawModifiedDetails.IsDeleteModified=true` (no modified-value history is kept); and the out-of-scope detail types — **event** HistoryUpdate (`UpdateEventDetails` / `DeleteEventDetails`) and `UpdateStructureData` — which are not overridden, so the SDK base rejects them. A backend exception is `BadInternalError` for that node and a deadline is `BadTimeout`. No `AuditHistoryUpdateEvent` is emitted. ### The `HistoryWrite` AccessLevel bit The SDK base admits a node to HistoryUpdate only if its `AccessLevel` carries `HistoryWrite`. The server ORs that bit in **only when the node is historized *and* an updater is wired at materialise time** (the raw node and every UNS mirror; the surgical `UpdateTagAttributes` path recomputes it the same way). `OtOpcUaServerHostedService.StartAsync` therefore wires the updater **before** installing the sink, and nodes materialised earlier are never retro-updated. With `Backend=Gateway` the bit is never set. ### Authorization HistoryUpdate is gated in **both** ACL modes (rewriting recorded history was never ungated). `WriteOperate` is deliberately **not** reused. | `Security:Acl:Enabled` | Decision | |---|---| | `false` (default) | The session's API key is re-checked first (`AuthorizeKeyUsable`, Gitea #563 — an expired or disabled key loses its activation-time roles), then the session must carry the **`HistoryUpdate` role**. `ApiKeyInterimRoleMapper` confers it **only from a `Cluster`-scoped grant** carrying `NodePermissions.HistoryUpdate`; narrower-scoped grants confer nothing (the same under-grant rule as `WriteOperate`). Denied ⇒ `BadUserAccessDenied`. A `WriteOperate`-only key is denied. | | `true` | `DataPlaneAuthorizer.Authorize(identity, OpcUaOperation.HistoryUpdate, nodeId, realm)` — the scoped `NodeAcl` grant decides per node and **replaces** the role check (never chained), so a session with no roles succeeds on the strength of a scoped grant. Denied ⇒ `BadUserAccessDenied` for that node. | `NodePermissions.HistoryUpdate` (bit 12) is authorable in the AdminUI ACL editor and is **not** part of any bundle (`ReadOnly` / `Operator` / `Engineer` / `Admin`) — it must be granted explicitly. See [security.md](security.md#data-plane-authorization). --- ## Authoring workflow 1. Open the equipment's **Tags** tab on `/uns/equipment/{id}`. 2. Create or edit the tag. - For **typed-editor drivers** (Modbus, S7, OpcUaClient, etc.): check the **Historize this tag** checkbox and, if needed, fill in the **Historian tagname (override)** textbox. - For **raw-JSON editors** (Galaxy): you can check the same first-class checkboxes (they appear below the JSON textarea), or add `"isHistorized":true` (and optionally `"historianTagname":"..."`) directly in the textarea. 3. Save and publish. The server rebuilds its address space; the node materialises with `Historizing=true` and the `HistoryRead` AccessLevel bit. 4. Confirm with Client.CLI `read` that the node's `Status` is `Good` and that the value is updating. Then issue a `historyread` to verify the historian connection returns data. ### Native-alarm historian opt-out (`alarm.historizeToAveva`) A tag carrying a native `"alarm"` object has a **separate** historian opt-out for its **alarm transitions** (distinct from tag-value historization). On the Tag modal, check or uncheck **Historize to AVEVA** in the alarm section. This maps to `alarm.historizeToAveva` (`bool?`) in the `TagConfig` JSON: - **Absent or `true`** (default) — the alarm's transitions are written to AVEVA Historian via `HistorianAdapterActor`. - **`false`** — the durable AVEVA write is suppressed for this alarm's transitions. The live `/alerts` feed and OPC UA condition events are unaffected. The gate is applied in `HistorianAdapterActor` using `is not false` semantics, matching the scripted-alarm `HistorizeToAveva` posture. See [ScriptedAlarms.md §Native driver alarms](ScriptedAlarms.md#native-driver-alarms-equipment-tag-path) and [AlarmTracking.md §Historian write-back](AlarmTracking.md#historian-write-back-non-galaxy-alarms) for the full alarm-historian routing. --- ## Client.CLI historyread examples The `historyread` command reads historical data from any node. Supply start and end times in ISO 8601 UTC form. See [docs/Client.CLI.md](Client.CLI.md) for the full flag reference. > **v3 Batch 4 NodeIds.** A historized tag is readable through **either** of its two NodeIds — > the Raw-namespace node (`s=`, e.g. `Plant/Modbus/dev1/Speed`) or a referencing > equipment's UNS-namespace node (`s=///`). Both register > the same historian tagname, so the returned series is identical. The `ns=N` index is assigned > at connect time per the namespace URI (`https://zb.com/otopcua/raw` / > `https://zb.com/otopcua/uns`) — resolve it from the server's `NamespaceArray` rather than > hard-coding it. The examples below show a Raw-namespace read (`ns=2` illustrative). ```bash # Raw history for a historized tag via its Raw-namespace NodeId (last 24 hours by default) otopcua-cli historyread \ -u opc.tcp://localhost:4840/OtOpcUa \ -n "ns=2;s=Plant/Modbus/dev1/Speed" \ --start "2026-06-13T00:00:00Z" --end "2026-06-14T00:00:00Z" # Same series via a referencing equipment's UNS-namespace NodeId, limited to 100 values otopcua-cli historyread \ -u opc.tcp://localhost:4840/OtOpcUa \ -n "ns=3;s=filling/line1/station1/Speed" \ --start "2026-06-13T00:00:00Z" --end "2026-06-14T00:00:00Z" \ --max 100 # 1-hour average aggregate otopcua-cli historyread \ -u opc.tcp://localhost:4840/OtOpcUa \ -n "ns=2;s=Plant/Modbus/dev1/Speed" \ --start "2026-06-13T00:00:00Z" --end "2026-06-14T00:00:00Z" \ --aggregate Average --interval 3600000 # Authenticated read (ReadOnly role or higher required) otopcua-cli historyread \ -u opc.tcp://localhost:4840/OtOpcUa \ -n "ns=2;s=Plant/Modbus/dev1/Speed" \ --start "2026-06-13T00:00:00Z" --end "2026-06-14T00:00:00Z" \ -U reader -P password ``` Supported `--aggregate` values: `Average`, `Minimum`, `Maximum`, `Count`, `Start`, `End`, `StandardDeviation` (aliases: `avg`, `min`, `max`, `stddev`/`stdev`, `first`, `last`). `--interval` is the processing interval in milliseconds (default 3600000 = 1 hour). The `Total` aggregate is served by the server's OPC UA HistoryRead endpoint for any conformant client (derived as time-weighted Average × interval-seconds — see "Total aggregate derivation" above), but is not exposed by this bundled CLI. --- ## Live /run gate The live read gate requires a reachable `ZB.MOM.WW.HistorianGateway` (VPN to `wonder-sql-vd03`) with the AVEVA Historian behind it healthy. Set `ServerHistorian:Enabled=true` with the correct `Endpoint` (`https://host:5222`) and supply `ServerHistorian__ApiKey` via the environment, then deploy and publish at least one historized Galaxy tag. The gate is operator-driven — it is not part of the local docker-dev rig. The gateway-backed driver also ships an env-gated live suite (`Category=LiveIntegration`); see the `HISTGW_GATEWAY_ENDPOINT` / `HISTGW_GATEWAY_APIKEY` / `HISTGW_TEST_TAG` / `HISTGW_WRITE_SANDBOX_TAG` / `HISTGW_BOOL_SANDBOX_TAG` / `HISTGW_ALARM_SOURCE` env vars (it skips cleanly when they are absent). `HISTGW_BOOL_SANDBOX_TAG` (e.g. `HistGW.LiveTest.BoolSandbox`, a **dedicated** writable Boolean tag — never the shared Float write sandbox) drives the Boolean historization leg (`EnsureTags_boolean_sandbox_provisions_as_Int2`) and the retype probe (`EnsureTags_type_change_outcome_is_observable`) that back the [Boolean historization maps to Int2](#boolean-historization-maps-to-int2-issue-441) decision. Running the full suite against the gateway (all six tests, none skipped) also discharges 06/U-2's "full documented run" for the historian suite. See [AlarmHistorian.md](AlarmHistorian.md) for the alarm write path and [ServiceHosting.md](ServiceHosting.md) for the (external) HistorianGateway deployment. --- --- ## Cross-driver TagConfig key reference The `TagConfig` JSON blob is the single extension surface for per-tag server-side behaviours across all drivers. Keys from multiple phases coexist in the same blob; unknown keys are preserved byte-stable on round-trip by all typed editors. ### Historian + alarm keys (Phase B / Phase C) | Key | Type | Phase | Description | |---|---|---|---| | `isHistorized` | bool | C | Marks the tag historized (HistoryRead + Historizing=true). | | `historianTagname` | string | C | Explicit historian tagname override (defaults to `FullName`). | | `alarm` | object | B | Native alarm definition (`alarmType`, `severity`, `historizeToAveva`, …). | ### Array keys (Phase 4c) | Key | Type | Phase | Description | |---|---|---|---| | `isArray` | bool | 4c | When `true` and `arrayLength >= 1`, materialises the node as a 1-D array (`ValueRank=OneDimension`). Absent or `false` → scalar. | | `arrayLength` | uint (≥1) | 4c | Element count; sets `ArrayDimensions`. Required when `isArray: true`. | ### Cross-driver array coverage matrix | Driver | Read mechanism | Live-verify status | |---|---|---| | **Modbus** | Contiguous FC03/FC04 block; String/BitInRegister modes supported | Mac-verifiable (sim `10.100.0.35:5020`) | | **S7** | `ReadBytesAsync` block + per-element decode loop | Unit-proven (fixture down) | | **AB CIP** | libplctag native array read (atomic + UDT member arrays) | Unit-proven (fixture down) | | **AB Legacy** | PCCC multi-element file read via libplctag (cap 256 elements) | Unit-proven (fixture down) | | **TwinCAT** | Raw-bytes array read + per-element decode (native notifications carry the whole array) | Live-verified on TwinCAT 2.11 (2026-10-07) | Array writes, multi-dimensional arrays (ValueRank>1), and per-element historization are out of scope — see [Uns.md §Array tags](Uns.md#array-tags-1-d). --- ## Closed backlog §2 items (Phase 4 / Phase 4b) The following items from the backlog §2 (archived at `docs/plans/2026-06-18-stillpending-backlog-audit.md`) were closed by earlier phases and are recorded here so future audits don't re-flag them. | Item | Closed by | Commit | |---|---|---| | Modbus Int64 / UInt64 OPC UA node `DataType` correction | Phase 4 | `bd8fee61` | | `HistoryAggregateType.Total` — `Total` aggregate | Phase 4 (derived client-side as time-weighted Average × interval-seconds) | `5e27b5f7` | | Historian poison alarm-event indefinite retry — dead-letter cap (task #437) | Phase 4 | `fcb38014` | --- ## See also - [docs/plans/2026-06-14-galaxy-phase-c-historian-design.md](plans/2026-06-14-galaxy-phase-c-historian-design.md) — full design and implementation notes - [AlarmHistorian.md](AlarmHistorian.md) — alarm write path; drains to the same HistorianGateway (`SendEvent`) - [AlarmTracking.md](AlarmTracking.md) — OPC UA Part 9 alarm surface (event history source) - [Client.CLI.md](Client.CLI.md) — full `historyread` flag reference - [ScriptedAlarms.md](ScriptedAlarms.md) §"Native driver alarms" — the Phase B `alarm` object in `TagConfig` (parallel carrier) - [Uns.md §Array tags](Uns.md#array-tags-1-d) — `isArray`/`arrayLength` keys, cross-driver coverage, and deferrals