lmxopcua/docs/Historian.md

# Historian — server-side OPC UA HistoryRead / HistoryUpdate (pluggable backends)

Phase C wires server-side OPC UA **HistoryRead** for authored equipment tags flagged
historized. The feature is driver-agnostic: any equipment tag (Galaxy, Modbus, OpcUaClient,
or any other driver) can be marked historized; the server dispatches all history reads to the
registered `IHistorianDataSource` — the **HistorianGateway** read client
(`GatewayHistorianDataSource`, talking gRPC to the external `ZB.MOM.WW.HistorianGateway` via the
`ZB.MOM.WW.HistorianGateway.Client` package). No EF migration is required; the historian flag rides in
the existing schemaless `TagConfig` JSON blob alongside the Phase B `alarm` object. (The bespoke
Wonderware TCP sidecar backend this replaced was retired — see
[drivers/Historian.Wonderware.md](drivers/Historian.Wonderware.md).)

> **The backend is pluggable (2026-10).** `ServerHistorian:Backend` selects `Gateway` (the default — everything
> described below as "the gateway"), `SqlServer` (a plain SQL Server database the server reads and writes
> itself, which also supports OPC UA **HistoryUpdate**) or `QuestDb` (a QuestDB time-series database over the Postgres
> wire protocol; also supports HistoryUpdate, via soft delete). See [Backend selection](#backend-selection-serverhistorianbackend),
> [SQL Server backend](#sql-server-backend), [QuestDB backend](#questdb-backend) and [HistoryUpdate](#historyupdate). Wherever this page says
> "the historian" without qualification it means the selected backend; sections that only apply to one backend say so.

Design references: [docs/plans/2026-06-14-galaxy-phase-c-historian-design.md](plans/2026-06-14-galaxy-phase-c-historian-design.md)
(Phase C) and [docs/plans/2026-10-09-pluggable-historian-design.md](plans/2026-10-09-pluggable-historian-design.md) (pluggable backends).

---

## Historized TagConfig schema

A tag is historized by setting the **Historize this tag** checkbox and optional **Historian
tagname (override)** textbox in the Tag modal on the `/uns` equipment page Tags tab. These
controls work for **all drivers** — typed editors (Modbus, S7, OpcUaClient, etc.) and the
raw-JSON textarea (Galaxy) alike. The controls merge `isHistorized` / `historianTagname`
into the existing `TagConfig` JSON blob via the `TagHistorizeConfig` helper, preserving all
other keys byte-stable. Drivers that still use the raw-JSON editor (Galaxy) can also add the
fields directly in the textarea.

### Fields

| Field | Type | Required | Description |
|---|---|---|---|
| `isHistorized` | bool | yes | Marks the tag historized. Materialises the OPC UA node with `Historizing=true` and the `HistoryRead` AccessLevel bit. |
| `historianTagname` | string | no | Explicit historian tagname to query. When absent or empty, defaults to the tag's driver `FullName` (the `TagConfig.FullName`). |

### Examples

Default historian tagname (uses `FullName` = `TestMachine_002.TestFloat`):

```json
{"FullName":"TestMachine_002.TestFloat","isHistorized":true}
```

Explicit historian tagname override:

```json
{"FullName":"TestMachine_002.TestFloat","isHistorized":true,"historianTagname":"Plant.Line1.Flow"}
```

A tag that is both historized and a native alarm:

```json
{"FullName":"TestMachine_002.HiAlarm","isHistorized":true,"alarm":{"alarmType":"OffNormalAlarm","severity":700}}
```

---

## Backend selection (`ServerHistorian:Backend`)

`ServerHistorian:Backend` (case-insensitive, trimmed; default `Gateway`) picks the provider
(`IHistorianBackendProvider`) that builds all five historian seams — the HistoryRead data source, the continuous-
historization value writer, the alarm-history writer, the tag provisioner and (optionally) the HistoryUpdate
updater. `HistorianBackendRegistry` in the Host owns the name table. An unknown name fails host start with an
`OptionsValidationException` that lists the allowed values (`Gateway`, `SqlServer`, `QuestDb`), regardless of whether anything
consumes the historian. The existing enable gates (`ServerHistorian:Enabled`, `AlarmHistorian:Enabled`,
`ContinuousHistorization:Enabled`) are unchanged — they simply get their objects from the selected provider.

| | `Gateway` (default) | `SqlServer` | `QuestDb` |
|---|---|---|---|
| Storage | AVEVA Historian via the HistorianGateway sidecar | `{Schema}` tables in a SQL Server database you point it at | `{prefix}sample` / `event` / `tag` / `sample_modified` WAL tables in a QuestDB instance |
| Value kinds recorded | `Double`, `Int64`, `Boolean` (numeric analog only) | all: `Double`, `Int64`, `Boolean`, `String`, `DateTime`, plus `Empty` (a Bad sample with no value) | same as `SqlServer` |
| Status code stored | Sent as DA quality but **discarded** by the gateway (C-002); AVEVA stores every sample Good | Stored bit-for-bit (`StatusCode` int column) | Stored bit-for-bit (unsigned, in a `LONG`) |
| Comms-loss / driver-Bad publishes | Dropped, counted in `DroppedUnsupportedKind` | Stored as an `Empty` outage-marker sample carrying the Bad code (see [below](#typed-values-real-status-and-outage-markers)) | Stored as an `Empty` outage-marker sample, as on `SqlServer` |
| Processed aggregates | Average, Minimum, Maximum, Total, Count | The seventeen in the [aggregate table](#aggregates-processed-reads) (incl. Part 13 time-weighted, #671) | The same seventeen, identical results |
| HistoryUpdate | No (`BadHistoryOperationUnsupported`) | Yes (Insert / Replace / Update / DeleteRaw / DeleteAtTime) | Yes — soft delete (tombstones), see [below](#historyupdate-soft-delete) |
| Modified-value history (`IsReadModified`, #668) | No (`BadHistoryOperationUnsupported`) | Yes — `SampleModified`, atomic with the change | Yes — `{prefix}sample_modified`, written before the change (orphan-on-crash) |
| Alarm history | gateway `SendEvent` | `Event` table (insert-if-absent) | `{prefix}event` table (dedup upsert on `(ts, event_id)`) |
| Provisioning (`EnsureTags`) | numeric types only; others skipped | `MERGE` upsert of every request, all types | Upsert into `{prefix}tag`, all types |
| Connection | `Endpoint` + `ApiKey` + TLS keys (top-level `ServerHistorian:*`) | `ServerHistorian:SqlServer:*` | `ServerHistorian:QuestDb:*` |

Recording stays **Primary-only** (#537) and **at-least-once** with the #538 per-tag quarantine on every backend;
HistoryRead and HistoryUpdate are served from any node (the database is shared). The historian connection string is
**not** a ConfigDb string, and a driver-only node may carry it without breaking the Phase 4 "no ConfigDb" posture.

---

## ServerHistorian configuration

The `ServerHistorian` section in `appsettings.json` controls the historian read path.
`Enabled` defaults to `false`; when disabled, the server registers `NullHistorianDataSource`
and all HistoryRead calls on historized nodes return `GoodNoData` (empty, not an error).

```json
{
  "ServerHistorian": {
    "Enabled": false,
    "Backend": "Gateway",
    "Endpoint": "",
    "ApiKey": "",
    "UseTls": true,
    "AllowUntrustedServerCertificate": false,
    "CaCertificatePath": null,
    "CallTimeout": "00:00:30",
    "MaxTieClusterOverfetch": 65536,
    "HistoryReadBatchParallelism": 4,
    "MaxConcurrentHistoryReadBatches": 16,
    "HistoryReadDeadline": "00:01:00",
    "MaxTimeWeightedRawRows": 1000000,
    "SqlServer": {
      "ConnectionString": "",
      "Schema": "hist",
      "CommandTimeout": "00:00:30",
      "AutoCreateSchema": true
    },
    "QuestDb": {
      "ConnectionString": "",
      "TablePrefix": "hist_",
      "RetentionDays": null,
      "CommandTimeout": "00:00:30",
      "AutoCreateSchema": true,
      "VisibilityTimeout": "00:00:05"
    }
  }
}
```

| Key | Type | Default | Description |
|---|---|---|---|
| `Enabled` | bool | `false` | Enable the selected backend's read client. `false` → `NullHistorianDataSource` (empty reads) and no HistoryUpdate. |
| `Backend` | string | `Gateway` | `Gateway`, `SqlServer` or `QuestDb` (case-insensitive). Unknown ⇒ host start fails. See [Backend selection](#backend-selection-serverhistorianbackend). |
| `Endpoint` | string | `""` | **Gateway backend only.** Absolute gateway URI, e.g. `https://host:5222`. Scheme selects transport (`https://` = TLS, `http://` = h2c plaintext). Required when `Enabled`. |
| `ApiKey` | string | `""` | **Gateway backend only.** The gateway peppered-HMAC key (`histgw_<id>_<secret>`) sent as `Authorization: Bearer`. Required when `Enabled`. **Supply via env `ServerHistorian__ApiKey`.** |
| `UseTls` | bool | `true` | **Gateway backend only.** Connect over TLS; must match the `Endpoint` scheme. |
| `AllowUntrustedServerCertificate` | bool | `false` | **Gateway backend only.** Accept a self-signed / untrusted server certificate (dev / on-prem only). |
| `CaCertificatePath` | string\|null | `null` | **Gateway backend only.** PEM CA file pinning the gateway's TLS chain. Null/empty uses the OS trust store. |
| `CallTimeout` | TimeSpan | `00:00:30` | **Gateway backend only.** Per-call deadline applied to each unary gateway read. A read for a tag the historian does not yet know about — e.g. a freshly-historized tag that has **not** been provisioned (see [Tag auto-provisioning](#tag-auto-provisioning-ensuretags)) — can run to this **full** deadline before the gateway errors, so an unprovisioned-tag HistoryRead can block for ~30 s. Lower this to fail faster, at the cost of truncating legitimately long reads. |
| `MaxTieClusterOverfetch` | int | `65536` | (Node-manager paging; all backends — a `SqlServer` PK rules out tie clusters, so it is moot there.) Maximum samples the server will fetch in one shot to page through a tie cluster (multiple samples sharing one `SourceTimestamp`). A cluster larger than this ceiling fails `BadHistoryOperationUnsupported`. Raise to handle abnormally large tie clusters; the default covers all normal-data cases. |
| `HistoryReadBatchParallelism` | int | `4` | Max historized nodes served **concurrently** within one HistoryRead batch (arch-review 03/S3). The SDK's `HistoryRead*` override surface is synchronous, so each arm bridges once at its boundary; this turns per-node serving from sequential (`N × CallTimeout` worst case) into `⌈N / P⌉ × CallTimeout`. Also caps the gateway-load multiplication a single client can command. `≤ 1` serves sequentially. |
| `MaxConcurrentHistoryReadBatches` | int | `16` | Process-wide limit on HistoryRead (and HistoryUpdate) batches served **concurrently** (arch-review 03/S3). A flood of history clients degrades to `BadTooManyOperations` rather than exhausting the SDK request-thread pool. With `HistoryReadBatchParallelism` this caps total in-flight gateway reads at `MaxConcurrentHistoryReadBatches × HistoryReadBatchParallelism` (default 16 × 4 = 64). A batch that cannot get a slot within `min(HistoryReadDeadline, 5 s)` fails every handle `BadTooManyOperations`. |
| `MaxTimeWeightedRawRows` | int | `1000000` | `SqlServer` / `QuestDb` only (#671). The most raw in-window samples one [time-weighted aggregate](#time-weighted-aggregates-part-13-671) read (TimeAverage, TimeAverage2, **Total**, Total2, Interpolative) may stream into the server-side evaluator — that family is **not** pushed down to SQL. The window query is capped at this + 1 rows; one row past the cap fails **that node** with `BadTooManyOperations` (not counted as a backend fault) instead of holding a pooled connection until `HistoryReadDeadline` and returning `BadTimeout`. Bound rows outside the window do not count; per-sample aggregates are not capped. 1 000 000 ≈ 11.5 days of a 1 Hz tag. Non-positive = unbounded (a `Validate()` warning). |
| `HistoryReadDeadline` | TimeSpan | `00:01:00` | Per-request wall-clock deadline for a HistoryRead (and HistoryUpdate) batch (arch-review 03/S3). A node still in flight at the deadline surfaces `BadTimeout`, so one request can never hold an SDK request thread longer than this regardless of node count. Non-positive = unbounded (a `Validate()` warning). |

**`ServerHistorian:SqlServer`** (read only when `Backend=SqlServer`; bound by `HistorianBackendRegistry.BindSqlServer`):

| Key | Type | Default | Validation | Description |
|---|---|---|---|---|
| `SqlServer:ConnectionString` | string | `""` | **Required** (non-blank) | ADO.NET connection string for the historian database. Never the ConfigDb. Supply via `${secret:OT-HISTORIAN-SQL}` / environment (`ServerHistorian__SqlServer__ConnectionString`) — never commit a real one. The database must already exist; `AutoCreateSchema` creates the *schema and tables*, not the database. |
| `SqlServer:Schema` | string | `hist` | Must match `^[A-Za-z_][A-Za-z0-9_]{0,127}$` (it is spliced into DDL/DML) | SQL schema that holds `SchemaVersion`, `Tag`, `Sample`, `Event` and the `SampleRow`/`TimeRow` table types. |
| `SqlServer:CommandTimeout` | TimeSpan | `00:00:30` | Must be `> 0` | Per-command timeout, rounded up to whole seconds (minimum 1 s). The SQL backend does **not** use `CallTimeout`. |
| `SqlServer:AutoCreateSchema` | bool | `true` | — | `true`: on first use apply any missing embedded script under a `sp_getapplock` (`OtOpcUaHistorianSchema`), so concurrent nodes serialise. `false`: only verify `SchemaVersion >= 1`; if behind, **every** historian call fails with a clear error naming the missing version (surfaced in the read client's health `LastError`, never a host crash). A failed bootstrap is retried on the next call, not latched. |
| `SqlServer:RetentionDays` | int? | unset | `>= 1` when set | Days of history kept, on **source time**. Unset keeps everything. When set, driver nodes purge older `hist.Sample` / `hist.Event` rows — see [Retention](#retention-retentiondays). |
| `SqlServer:PurgeInterval` | TimeSpan | `01:00:00` | `> 0` | How often the purge runs (first pass at start-up). |
| `SqlServer:PurgeBatchSize` | int | `4000` | `>= 1` | Rows per `DELETE TOP (n)`; the default stays under SQL Server's ~5000-lock escalation threshold so a purge never takes a table lock. |

**`ServerHistorian:QuestDb`** (read only when `Backend=QuestDb`; bound by `HistorianBackendRegistry.BindQuestDb`): six keys — `ConnectionString`
(required), `TablePrefix`, `RetentionDays`, `CommandTimeout`, `AutoCreateSchema`, `VisibilityTimeout`. The full table, defaults and validation
rules are in [QuestDB backend → Configuration](#configuration-serverhistorianquestdb) (and [Configuration.md](Configuration.md#serverhistorianquestdb-backendquestdb)).

**Validation rules** (`ServerHistorianOptionsValidator`, driver-role nodes only; admin-only nodes never build historian singletons):

- `Backend` must name a registered backend — checked **always**, with no I/O, even when nothing is enabled.
- Everything else is **consumer-gated**: it runs only when `ServerHistorian:Enabled` **or** `AlarmHistorian:Enabled` is true
  (the alarm sink sources its connection from this section even when `ServerHistorian` is off).
  - `Backend=Gateway`: `Endpoint` must be an absolute http(s) URI, `ApiKey` non-empty, and `UseTls` must agree with the endpoint scheme.
  - `Backend=SqlServer`: the three `SqlServer` rules above. `Endpoint`/`ApiKey`/`UseTls` are **not** required.
  - `Backend=QuestDb`: `QuestDbHistorianOptions.Validate()` (connection string required, prefix regex, `RetentionDays >= 1`, `CommandTimeout > 0`,
    `VisibilityTimeout >= 0`). `Endpoint`/`ApiKey`/`UseTls` are **not** required. A malformed `TablePrefix` is also rejected by the connection helper
    when the provider first builds a seam.
- ⚠️ `Backend=SqlServer` with a malformed `Schema` throws `ArgumentException` while the provider is built, so it stops
  host start even when no consumer is enabled (the validator is consumer-gated, provider construction is not).
- The alarm-only warning at startup (`AlarmHistorian` on, `ServerHistorian` off) names the selected backend.

> **Do not commit `ApiKey` to `appsettings.json`.** Set it via the environment variable
> `ServerHistorian__ApiKey`, a secrets store, or a deployment-time overlay. The checked-in default is
> always empty.

> **Gateway-side prerequisites.** The target gateway must run `RuntimeDb:Enabled=true` (continuous
> `WriteLiveValues`) + `RuntimeDb:EventReadsEnabled=true` (alarm-history `ReadEvents`), and the API key
> must carry the scopes `historian:read`, `historian:write`, `historian:tags:write`.

> **Migration from the Wonderware backend.** Rename the old keys: `Host`/`Port` → `Endpoint`
> (`https://host:5222`); `SharedSecret` → `ApiKey` (env `ServerHistorian__ApiKey`);
> `ServerCertThumbprint` → `CaCertificatePath` (+ `UseTls` / `AllowUntrustedServerCertificate`).

The `ServerHistorian` section is independent of the `AlarmHistorian` section (the alarm
write path) and the `ContinuousHistorization` section (driver-value capture). All three target the
**same** gateway — but only `ServerHistorian` carries the connection (endpoint/key/TLS); the other two
source it from there.

> **Continuous-historization value quality on the Gateway backend is still server-stamped.** The recorder now
> carries each sample's **real** OPC UA status (it used to hard-code Good `192`, so a Bad sample was historized as
> Good). `GatewayHistorianValueWriter` maps that status to an OPC-DA quality byte with
> `GatewayQualityMapper.ToDaQuality` (the inverse of the read-side table; unknown codes fall back by severity:
> Good ⇒ 192, Uncertain ⇒ 64, Bad ⇒ 0). But the gateway's `WriteLiveValues` SQL path **discards** the quality
> it is sent (a server-managed column; 0.2.0 Contracts C-002), so **AVEVA still stores every recorded value as
> Good** — a pre-existing limitation of that backend, now documented rather than fixed (archreview 06/C-7). The
> `SqlServer` backend stores the real status.

> ✅ **Continuous value capture is LIVE-PROVEN END-TO-END
> ([#491](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/491) CLOSED — gate run 2026-08-01,
> completed 2026-08-04; see `docs/plans/2026-08-01-continuous-historization-live-gate.md`).** Against a
> real `ZB.MOM.WW.HistorianGateway` talking to the real AVEVA Historian on `wonder-sql-vd03`: deploy →
> `EnsureTags` provisioning → dependency-mux value delta → FasterLog outbox append → drain →
> `WriteLiveValues` → **value read back from the historian**, plus **restart convergence** (the recorder
> re-registers the deployed ref set after a process restart, with no new deployment).
>
> The read-back used two registers carrying **distinct non-zero** constants (`8224`, `120`), each of
> which read back as its own value — a `0`-valued source cannot distinguish a correct value from a
> wrong one.
>
> ⚠️ The gate found a **production-fatal defect**: the recorder threw `NotSupportedException` on the
> **first value it ever appended**, because it touched `Self` after an `await …ConfigureAwait(false)`,
> which discards Akka's ActorContext. Supervision restarted the actor, and since the recorder is seeded
> with an *empty* ref set, the restart discarded every historized ref — historization stopped after one
> value. Fixed; pinned by a regression test whose outbox fake completes on a **pool** thread (a fake
> returning `ValueTask.CompletedTask`, or even `await Task.Yield()`, cannot reproduce it).
>
> The gate also found the two defects below, both since fixed.
>
> **Related, separately open:** `EnsureTags` provisioning fires only for **newly-added** historized
> tags (`AddressSpaceApplier` iterates the added set), so flipping an **existing** tag to
> `"isHistorized": true` never provisions it — see
> `docs/plans/2026-06-27-historian-gateway-integration-issues.md` Issue 5.

---

## ContinuousHistorization configuration

`ContinuousHistorizationOptions` binds the `ContinuousHistorization` section. It carries **only** the
recorder + outbox knobs — the backend connection (gateway endpoint / key / TLS, or the SQL Server connection
string) is sourced from `ServerHistorian` (above), never restated here. The outbox is a node-local FasterLog
and is registered the same way for every backend.

| Key | Type | Default | Description |
|---|---|---|---|
| `Enabled` | bool | `false` | `true` **and** a configured `ServerHistorian` wires the FasterLog outbox + the selected backend's value writer and spawns the `ContinuousHistorizationRecorder`. `false` → no recorder; driver tag values are not historized. |
| `OutboxPath` | string | `""` | **Directory** holding the FasterLog segment + commit files. Required when `Enabled`. In production set an **absolute** path on durable storage — a relative path resolves against the host's working directory. |
| `CommitMode` | string | `PerEntry` | `PerEntry` fsyncs before the append returns (no loss window); `Periodic` batches commits onto a background timer every `CommitIntervalMs`. Parsed case-insensitively; an unrecognized value falls back to `PerEntry`. **See the batch-commit note below — `PerEntry` is one fsync per ingest *burst*, not per value.** |
| `CommitIntervalMs` | int | `100` | Periodic-mode commit cadence (ms); must be `> 0` under `Periodic`, ignored under `PerEntry`. |
| `DrainBatchSize` | int | `1024` | Outbox entries peeked + written per drain pass. **Raised from 64**: the peek is an in-memory scan of the outbox's live index (widening it costs allocation, not I/O), and a pass is now bounded-parallel over `DrainConcurrency` rather than sequential, so a wide batch no longer costs `tags × latency`. At 64 a drain drew down at most 64 entries every `DrainIntervalSeconds` — below the arrival rate of even a modest historized tag set — so any gateway hiccup left the outbox permanently behind, growing toward `Capacity` and shedding oldest samples instead of catching up. |
| `DrainIntervalSeconds` | double | `2` | Steady drain cadence (and the post-success reschedule). |
| `DrainConcurrency` | int | `8` | Historian tags a single drain pass writes **concurrently**. One peeked batch is grouped per historian tag and each group is an independent gateway round-trip, so a sequential pass cost `tags × latency`; this window makes it `⌈tags / concurrency⌉ × latency` while bounding how many round-trips can be outstanding against the gateway at once. `1` restores the fully sequential drain; a value `< 1` is clamped to 1 (with a `Validate()` warning). **The ack is unchanged** — still all-or-nothing after every group has reported, and the #538 per-tag quarantine still keys on "another group wrote this pass". |
| `Capacity` | int | `500000` | Max un-acked outbox entries before the drop-oldest policy sheds the front of the queue; `0` = unbounded. Roughly 50 MB of FasterLog. The #538 **disk backstop** for the one case the per-tag quarantine deliberately cannot resolve — a historian unreachable or refusing *every* write, where nothing proves any particular tag is at fault so the recorder correctly retains. |
| `MaxWriteAttempts` | int | `5` | #538 poison-tag budget: consecutive failed drain passes one historian tag may accumulate before its queued entries are abandoned and the FIFO head is allowed to advance. **Counted only across passes where *other* tags wrote successfully**, so an ordinary gateway outage never advances it. Clamped to at least 1; each abandonment logs a Warning naming the tag. |
| `CaptureWhilePrimaryUnreachable` | bool | `true` | #672: a standby whose capture is stood down for its mesh-partner Primary (#537) resumes capturing as soon as the cluster failure detector reports that Primary **unreachable**, instead of waiting for auto-down + a fresh redundancy snapshot. Shrinks a crash-failover capture gap from ~25 s to the failure-detector window (~10–12 s), at the cost of short duplicate windows — see [the crash-failover note](#a-crashed-primary-capture-gap-672). `false` restores the pre-#672 behaviour. Never affects the alarm drain or the Primary data-plane gate. |
| `MinBackoffSeconds` | double | `1` | Initial retry backoff after a failed drain pass. |
| `MaxBackoffSeconds` | double | `30` | Cap on the exponential retry backoff. |

> **`PerEntry` now means one fsync per ingest *burst*, not per value.** The recorder appends a whole
> value burst through `IHistorizationOutbox.AppendBatchAsync`, which enqueues every entry and then
> issues a **single** group commit — so a burst of N values costs one fsync rather than N. The
> durability contract is unchanged: the call does not return until that commit completes, so nothing
> the caller has been told is buffered can be lost to an unclean restart. (The documented residual is
> unchanged too: a throw *before* the commit leaves the enqueued prefix tracked-but-uncommitted —
> served correctly for the life of the process, absent after an unclean restart — exactly the
> enqueue-then-commit window the single-entry path always had, widened from one entry to the batch.)

> **Cross-repo follow-up — a multi-tag batch `WriteLiveValues` on the gateway contract.** The drain's
> unit of work is one gateway round-trip **per historian tag**, because
> `ZB.MOM.WW.HistorianGateway.Client` **0.3.0** exposes no RPC that writes values for more than one tag
> in a call. Bounded parallelism (`DrainConcurrency`) is therefore the ceiling: it overlaps the
> round-trips but cannot remove them, so a wide tag set still pays `⌈tags / concurrency⌉` network
> hops per drain pass. Collapsing that to a single hop needs a **batch/multi-tag `WriteLiveValues`** on
> the `historian_gateway.v1` contract in the `~/Desktop/HistorianGateway` sister repo. Until that
> lands, raising `DrainBatchSize` widens each hop's payload but not the hop count — which is why the
> concurrency window, not the batch width, is what governs drain throughput today.

### Typed values, real status and outage markers

The recorder no longer assumes "a numeric Good sample". Each mux delta is coerced to a typed `HistorianValue`
(`Double` / `Int64` / `Boolean` / `String` / `DateTime` / `Empty`; a `ulong` above `long.MaxValue` becomes `Double`;
arrays and anything else are dropped and counted) and appended with the **real** status the driver published.
A value whose kind is not in the selected backend's `SupportedKinds` is dropped at capture and counted in
`RecorderStatus.DroppedUnsupportedKind` (the field was `DroppedNonNumeric` before the rename; `Empty` is supported
by `SqlServer`, not by `Gateway`).

⚠️ **Comms loss is historized — as an outage marker (design §9.10).** `DriverHostActor.FanCommsLossToDependencyMux`
(#481) fans `(null, BadNoCommunication)` for every value ref of a comms-lost driver to the dependency mux, and the
recorder cannot tell that from a driver's own Bad publish. So:

- **`SqlServer` and `QuestDb`:** each disconnect stores **one `Empty` / `BadNoCommunication` (`0x80310000`) sample per historized
  tag**. A driver's own Bad publish is stored the same way (`Empty` kind plus its own code — e.g. Modbus
  `BadCommunicationError` `0x80050000`). This is deliberate: raw reads show the gap, and aggregates and AtTime
  interpolation do not bridge it. Cost: one extra row per tag per outage.
- **`Gateway`:** the marker is `Empty`, which that backend does not support, so it is dropped and counted in
  `DroppedUnsupportedKind`.

### Outbox format v2 and rollback

Outbox entries are written in format **v2**: `"HV2\0"` magic + version byte `2` + id + tag + value kind + a
kind-specific payload + the 32-bit status + timestamp. Entries left in an outbox by an older build (the fixed
38-bytes-plus-tag layout, always quality 192) are still **read** — as a `Double` with status `Good`. A record that
matches neither layout is **skipped and counted, never thrown**, so a corrupt record cannot jam the drain (this also
holds during the FasterLog recovery scan at start-up); the lifetime count is the gauge
**`otopcua.historian.outbox.unreadable`** (`IHistorizationOutbox.UnreadableCount`). A growing value means
undecodable records are being skipped.

> **Metric prefixes differ.** The outbox's other two gauges are `otopcua.historization.outbox.depth` and
> `otopcua.historization.outbox.dropped`; the unreadable gauge is **`otopcua.historian.outbox.unreadable`**
> (the name the design fixed). All three describe the same outbox — search for both prefixes.

⚠️ **Rollback: drain or delete the outbox before downgrading.** A node running a pre-v2 build cannot read v2
records — its outbox deserializes every record in its constructor and fails to start on the first v2 one. Before
rolling a node back, let the outbox drain to zero (`otopcua.historization.outbox.depth`) or stop the node and
delete the directory named by `ContinuousHistorization:OutboxPath` (accepting the loss of anything un-drained).
Upgrading needs no action.

**Delivery semantics on `SqlServer`** (`QuestDb` is the same, at µs resolution and with tombstones counting as present — see [QuestDB backend](#live-writes-insert-if-absent-tombstones-included)). Writes are *insert-if-absent* on `(TagId, SourceTimeUtc)`: the outbox's
at-least-once redelivery (and both pair nodes writing the same sample in a boot window) converges on one row, and the
first-stored sample wins. **Duplicate timestamps inside one batch collapse to the first.** A `NaN` / `±Infinity` `Double`
cannot be stored in a SQL `float`, so the **value writer stores it as an `Empty` sample** — the tag is not quarantined for
one bad reading (HistoryUpdate behaves differently: see [below](#historyupdate)). ⚠️ **An `Empty` sample is always Bad**
(the quality contract: a stored sample is never served better than "no value"): a non-finite `Double` that arrived with a
Good or Uncertain status is stored as `Empty` + **`BadOutOfRange`** (`0x803C0000`), and any other `Empty` with a non-Bad
status as `Empty` + **`BadNoData`** (`0x809B0000`). An existing Bad code — the comms-loss `BadNoCommunication` marker, a
driver's own Bad publish — is stored as it is. A duplicate-key
error (2627/2601) rolls the whole statement back, so the writer re-runs the statement (at most twice) instead of
treating the error as success.

---

## A poison tag no longer pins the outbox (#538)

`DrainOnceAsync` peeks a batch, writes **every** per-tag group in it, and then acks **all-or-nothing** —
because the outbox's `RemoveAsync` truncates the FIFO head *through* the acked id, so acking one tag's
entries could silently drop older, unwritten entries belonging to another. That much is deliberate and
unchanged.

What was wrong is what happened when a tag failed **permanently**: the head never advanced, so every
**healthy** tag in the batch was re-submitted to the historian on every drain pass, indefinitely —
duplicate samples at identical timestamps, and an outbox that never truncated (observed live at 8.3 MB
from a single unprovisionable tag, with the gateway's `ok` **and** `error` counters climbing together —
the signature of this bug).

The drain now tracks **consecutive failures per historian tag** and abandons a tag's queued entries once
it exceeds `ContinuousHistorization:MaxWriteAttempts` (default 5), letting the head advance. Two
properties matter:

- **A pass where NOTHING wrote abandons nothing.** The counter only advances on a pass where at least one
  *other* tag wrote successfully — that is what distinguishes "this tag is poison" from "the backend is
  down". A count-only quarantine looks identical on the poison-tag case and silently discards the whole
  backlog during an ordinary gateway restart, turning a duplicate-samples bug into a lost-history one.
  Pinned by a falsification-control test (`A_backend_that_fails_every_tag_abandons_nothing`) that goes red
  when the evidence check is removed while the poison-tag test stays green.
- **A tag over budget waits for the others.** Abandonment is deferred until *every* failing tag in the
  batch is over budget, so a tag is never discarded on a pass that could not have advanced the head anyway.

Each abandonment logs a Warning naming the tag and the discarded sample count, and the total rides
`RecorderStatus.AbandonedEntries`. The most common cause is a tag the historian has no definition for —
easy to reach, because `EnsureTags` provisioning runs only for newly-**added** historized tags, so a
changed `historianTagname` is never provisioned (Issue 5 above).

`ContinuousHistorization:Capacity` now also defaults to **500,000** entries rather than unbounded. It is
the disk backstop for the one case the quarantine deliberately cannot resolve — a historian refusing
*every* write, where retaining is correct. Set `0` to restore unbounded retention.

## Only the recording Primary captures (#537)

Both nodes of a redundant pair serve the same address space and see the same dependency-mux fan-out, so
an ungated recorder wrote **every sample twice** (measured: the gateway's write counter advanced twice
per value with both centrals up, and halved when one was stopped).

The recorder now reads a capture gate per value, fed from `PrimaryGatePolicy.ShouldRecordHistorizedValues`
via `IHistorizationRoleView` — the same shape as the alarm store-and-forward sink's `drainGate`, with two
deliberate differences:

- **It gates the CAPTURE, not the drain.** The alarm queue lives in LocalDb and *replicates*, so the
  Secondary already holds the Primary's rows and gating the drain loses nothing. The historization outbox
  is a node-local FasterLog replicating to nobody: gating only its drain would leave the standby appending
  forever to a queue no one empties — unbounded disk, and a flood of stale duplicates the moment it is
  promoted. Entries a node already captured are still drained by that node, so a demotion loses nothing
  recorded.
- **The relevant peer is one that records the same refs**, not one that holds the same rows. A node stands
  down only for a Primary in its own per-cluster mesh. A node carrying no cluster-scoped role cannot
  establish that, so it never stands down: under a fleet-wide election the elected Primary is usually
  running a different cluster's configuration, and deferring to it would end this cluster's historization
  silently — the drain would report success on an empty outbox forever. That is the same scope mismatch
  that once suspended the alarm drain on every node of the docker-dev rig at once.

An unknown role **captures**, matching the alarm drain: a duplicate sample is recoverable, a hole in the
trend is not. The residual cost is a capture gap across a failover, bounded by how long the newly promoted
node takes to see its first redundancy snapshot — inherent to any non-duplicating design. Suppressed
values are counted in `RecorderStatus.SuppressedNotPrimary`, and each gate transition logs once.

### A crashed Primary: capture gap (#672)

On a **crash** (SIGKILL / `docker kill`) that "first snapshot" can be a long way off. The dead Primary
stays a cluster member — *unreachable, not downed* — for the failure-detector window plus
`auto-down-unreachable-after` (`acceptable-heartbeat-pause` 10 s + 15 s in `akka.conf`), and the
redundancy singleton almost always lived on it (it is the oldest member), so no snapshot is published
until auto-down re-homes the singleton. The standby's cached snapshot keeps naming the dead node Primary
and **nobody captures** for ~30 s (measured on the QuestDB live gate).

`DriverHostActor` therefore follows cluster reachability (its own `ClusterEvent.ReachabilityEvent`
subscription) and, with `ContinuousHistorization:CaptureWhilePrimaryUnreachable` on (the default),
**excludes an unreachable peer from the historization "a mesh-partner Primary is recording" question**,
re-deriving and republishing only the capture verdict from the last snapshot. A node at the same address
coming Up again clears the mark; a reachable-again Primary closes the gate again. It logs
`capturing historized values while recording Primary <id> is unreachable` once on entry and once on
revert. The alarm drain and the Primary data-plane verdict (writes, native acks, the Sparkplug host claim)
are **not** touched — a false allow there is a dual claimant, not a duplicate sample.

Trade-offs, all in the duplicate direction #537 already accepts:

- **The gap shrinks to the failure-detector window** (~10–12 s), not to zero — detection is the floor.
- **Duplicates during the overlap.** A Primary that is only *briefly* unreachable (GC pause, network blip)
  overlaps the standby until it is reachable again, and under a genuine **partition** both sides capture
  from detection onward (as they already did after auto-down, now ~15 s earlier). SqlServer and QuestDB
  absorb a duplicate of a **source-timestamped** sample idempotently (`(TagId, SourceTimeUtc)` key /
  dedup), but a driver with **no source timestamp** (e.g. Modbus: `SourceTimestampUtc` is null, so each
  node stamps its own poll time) writes a duplicate row on **every** backend, and the **Gateway** backend
  stores duplicates of everything.
- Set `CaptureWhilePrimaryUnreachable=false` to keep the pre-#672 behaviour (no duplicates from this
  path, the full ~30 s hole on a crash).

---

## Tag auto-provisioning (EnsureTags)

When a deploy adds a historized tag, the server auto-ensures it exists in the historian via the gateway's
`EnsureTags` (Gateway) or the `Tag`-row `MERGE` (SqlServer) **before** the first value or HistoryRead lands. `AddressSpaceApplier.Apply()` fires a
**non-blocking, fire-and-forget** `IHistorianProvisioning.EnsureTagsAsync` for each added historized value
tag (with the resolved historian name — the `historianTagname` override when set, else the driver
`FullName`); the hook is wrapped so a faulted, slow, or throwing provisioner can **never** block or fail a
deploy. On the **Gateway backend**, non-numeric (`String`/`DateTime`/`Reference`) types are skipped, not provisioned;
the SqlServer backend provisions every type (below).

**Wiring + gating.** The concrete provisioner comes from the selected backend's provider (`CreateProvisioning`):
the gateway-backed `GatewayTagProvisioner`, or — for `SqlServer` — `SqlServerTagProvisioner` (one `MERGE … WITH (HOLDLOCK)`
upsert per chunk of 400 tags; every data type is provisionable so `skipped` is always 0; a blank or over-400-character
name fails alone; a null unit/description in a request keeps the stored value). It is registered by the Host's
`AddHistorianProvisioning` and gated on the **same**
`ServerHistorian:Enabled` flag as the read path — both use the one resolved backend. When the section is
disabled (or absent) the applier resolves the no-op `NullHistorianProvisioning` and **provisioning does not
run** (so on a read-only-historian deployment, ensure tags exist by some other means). On the Gateway backend the
API key must carry the `historian:tags:write` scope for `EnsureTags` to succeed.

**Observability.** Every dispatch logs an Information tally —
`historian provisioning completed (dispatched=N, requested=…, ensured=…, skipped=…, failed=…)`. A line
showing `dispatched=N` but `requested=0` indicates the **no-op** provisioner ran (provisioning is not wired
for this node), which is the signal to check that `ServerHistorian:Enabled=true`.

> **Why this matters for HistoryRead latency.** Until a tag is provisioned, the historian does not know it,
> so a HistoryRead on it runs to the full `CallTimeout` (~30 s) before erroring. Auto-provisioning on deploy
> keeps freshly-historized tags from hitting that slow path. See the `CallTimeout` row above.

### Deferred types and the UInt16 fallback (#600 item 4)

> **Gateway backend only.** The `SqlServer` backend provisions and records every type (`String` and `DateTime` included).

Historization on the **Gateway** backend is **numeric-analog only**. `HistorianTypeMapper` classifies exactly nine types as
historizable (`Boolean`, `Int16/32/64`, `UInt16/32/64`, `Float32`, `Float64`):

- **Deferred: `String`, `DateTime`, `Reference`.** The historian's SQL analog write path cannot carry them.
  Provisioning skips them (counted in the `skipped=` tally, via `IsHistorizable`), the mapper throws
  `NotSupportedException` for them, and `ContinuousHistorizationRecorder` drops their values and counts them
  in `DroppedUnsupportedKind` (renamed from `DroppedNonNumeric`). A tag authored `isHistorized` with one of these types therefore silently records
  nothing; HistoryRead on it returns whatever the historian holds (typically `GoodNoData`).
- **Fallback: `UInt16 -> Uint4`.** The historian's `UInt2` write path is unproven upstream, so UInt16 tags
  are provisioned as the wider `UInt4`. Values are unaffected; only the historian-side storage type differs.
  (`Boolean -> Int2` is the other fallback, below.)

Lifting either needs gateway-side support first; neither is scheduled.

### Boolean historization maps to Int2 (issue #441)

`EnsureTags` is contract-documented (HistorianGateway 0.2.0+) as an **upsert** — *"creates or updates
historian tags according to the given definitions"* — that maps per-tag onto AVEVA's
`HistorianClient.EnsureTagAsync(definition) -> bool`. It is **not idempotent** (not routed through the
idempotent retry pipeline) and **not skip-if-exists**.

`HistorianTypeMapper` maps **`DriverDataType.Boolean → HistorianDataType.Int2`** and Boolean values
historize as a small **0/1 integer**. Int2 is used rather than the semantically-exact `Int1` because the
historian's **`Int1` analog-tag creation path is server-degenerate** — `EnsureTags(Int1)` stores an
unusable stub, so a Boolean tag mapped to `Int1` never becomes historizable. This is an evidence-based
decision from a live probe against the deployed 2023 R2 historian; the root cause and gateway-side
resolution are tracked in [issue #441](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/441) and
HistorianGateway [#11](https://gitea.dohertylan.com/dohertj2/historiangw/issues/11) / PR #16 (which
documents `Int1` as an explicit, evidence-based exclusion). The value-write path is unaffected: the
recorder sends every value as a `double` on `WriteLiveValues` regardless of tag type, so a Boolean
`0.0/1.0` round-trips cleanly into the Int2 tag.

**Observed live (2026-07-13, real 0.2.0 gateway → real AVEVA historian on `wonder-sql-vd03`, probed via
`grpcurl` — archreview R2-06 T12):**
- `EnsureTags` is a **real create-or-update upsert** — re-`EnsureTags` of an existing tag at the same type
  returns `Success=true`.
- An **in-place retype between SUPPORTED types is ACCEPTED** — `Float→Double` on an existing tag returns
  `Success=true`.
- A type sweep confirmed **`Int2/UInt2/Int4/UInt4/Float/Double` all provision `Success=true`; only `Int1`
  fails** (`Success=false, Error="ProtocolEvidenceMissingException"` — the vendored client's "no capture
  evidence for this type" guard → gRPC `Unimplemented`), both for a fresh tag and as a retype target. Mapping
  Boolean to `Int2` sidesteps that gap entirely.

> **Prerequisite for provisioning to land at all:** the provisioner must stamp a **positive `StorageRateMs`**
> — a `0` made `EnsureTags` throw `ArgumentOutOfRangeException` and fail *every* provision (fixed in
> `GatewayTagProvisioner`, PR #439).

**Migration note for pre-existing Boolean tags.** A Boolean tag historian-side as Float (e.g. from the
0.1.0-era default) is asked, on its next deploy, to upsert to Int2. Both are supported analog types, so the
retype is accepted. Provisioning is fire-and-forget and never blocks/fails a deploy; a per-tag rejection (if
any) surfaces only as the tally's `failed` count, and values written to a still-Float tag remain readable
(Float supersets the 0/1 range) pending reconcile. A bulk retype sweep is **not** done in code.

---

## HistoryRead behavior

### Read variants

The server supports all four OPC UA HistoryRead variants:

| Variant | Node type | CLI `--aggregate` |
|---|---|---|
| **Raw** | Historized variable | (omit `--aggregate`) |
| **Processed** | Historized variable | `Average`, `Minimum`, `Maximum`, `Count`, `Start`, `End`, `StandardDeviation` (the CLI's own list; `SqlServer` / `QuestDb` serve seventeen aggregates — see the [aggregate table](#aggregates-processed-reads)) |
| **AtTime** | Historized variable | n/a (client supplies exact timestamps) |
| **Events** | Equipment folder (event notifier) | n/a |
| **Modified** (Raw with `IsReadModified = true`, #668) | Historized variable | `--modified` (see [Modified-value history](#modified-value-history-part-11)) |

**Variable nodes** (historized tags) serve Raw, Processed, and AtTime history.
`Historizing=true` and `AccessLevels.HistoryRead` are set at materialization so any compliant
OPC UA client can discover historized capability from the node's attributes.

> **v3 Batch 4 — dual-namespace registration.** A historized raw tag surfaces as **two**
> variable nodes: the Raw-namespace node (`ns=Raw, s=<RawPath>`) and, for each referencing
> equipment, a UNS-namespace node (`ns=UNS, s=<Area>/<Line>/<Equipment>/<EffectiveName>`).
> **Both NodeIds register the SAME historian tagname** (the raw tag's `FullName` /
> `historianTagname`), so HistoryRead against either NodeId resolves to the same tagname and
> returns the same series. The historization intent is single-sourced — the mux's
> `HistorizedTagRef` set stays keyed by RawPath (no doubles), and continuous historization /
> `EnsureTags` provisioning run once per raw tag regardless of how many equipment reference it.

**Equipment-folder event-notifier nodes** serve Event history. Every equipment folder that
owns at least one alarm condition is already an event notifier; the server registers a
`sourceName` (the equipment id) for each such folder and maps event history reads to the
HistorianGateway using that source. (Alarm-history `ReadEvents` requires the gateway running
`RuntimeDb:EventReadsEnabled=true`.) Event-field projection supports the standard
`BaseEventType` select clauses — `EventId`, `SourceName`, `Time`, `ReceiveTime`, `Message`,
and `Severity`; an unsupported select operand returns a null field (spec-conformant).

### No driver-side history (Gitea #655)

History is served **only** by the server-wide `IHistorianDataSource`
(`GatewayHistorianDataSource` / `NullHistorianDataSource`). No driver contributes
history: the OpcUaClient driver does **not** proxy its upstream server's
HistoryRead (the former `IHistoryProvider` passthrough was never dispatched to and
was deleted). On an OpcUaClient tag, `isHistorized` means "OtOpcUa records this
value into the historian from now on", not "forward history reads upstream". A
node without `isHistorized` has no HistoryRead access bit and answers
`BadHistoryOperationUnsupported`.

### Graceful degradation

| Situation | HistoryRead status |
|---|---|
| Historized node, historian configured and reachable, results found | `Good` |
| Historized node, historian configured, time range empty | `GoodNoData` |
| Historized node, historian NOT configured (`Enabled=false` / Null source) | `GoodNoData` (empty) |
| Non-historized node | `BadHistoryOperationUnsupported` |
| Backend timeout or exception | `BadHistoryOperationUnsupported` per node; other nodes in the same batch are unaffected |
| `Backend=SqlServer`, `AutoCreateSchema=false`, schema missing/behind | every read/write fails with a clear error naming the missing schema version (health `LastError`); reads answer `BadHistoryOperationUnsupported` per node; no host crash |

A historized node with no historian configured never returns an error status — it returns
empty. This means a deployment can author and publish historized tags before the historian
gateway is provisioned, without producing error spikes in connected clients.

### Continuation-point paging (Raw)

`HistoryRead-Raw` is paged server-side. The historian backend is single-shot (it returns up to
`NumValuesPerNode` samples with no continuation point of its own), so the server synthesises
paging time-based:

- A page that returns **exactly** `NumValuesPerNode` samples (with `NumValuesPerNode > 0`) MAY
  have more behind it, so the server stores a resume cursor and returns an opaque continuation
  point (16 bytes). The client hands it back to fetch the next page.
- A short page (fewer than the cap) is the last page — no continuation point.
- `NumValuesPerNode == 0` ("all values, no limit") is never paged; the whole window returns in one
  shot.
- The resume cursor is **tie-safe**: the next page resumes from the last returned sample's
  SourceTimestamp *inclusive* and drops the boundary samples already emitted, so samples sharing
  the boundary timestamp are neither duplicated nor skipped.

> **Oversized tie clusters — within-timestamp paging.** When more samples share one
> `SourceTimestamp` than the current page cap, the server detects that the cursor has stalled on
> a tie cluster (the last returned timestamp equals the resume timestamp). It then **over-fetches
> the entire cluster** at that single timestamp up to a bounded ceiling controlled by
> `ServerHistorian:MaxTieClusterOverfetch` (default **65 536**), then serves the cluster
> `NumValuesPerNode` samples at a time across successive pages, advancing the cursor one tick past
> the timestamp once the cluster is fully drained.
>
> **Reverse reads** (`start > end`, newest first) page the same way in the opposite direction: a drained
> cluster steps the cursor one tick **down**, the read ends when that would reach the (excluded) `end`, and
> the over-fetched cluster is sliced in descending order. Before the fix a reverse read with a small
> `NumValuesPerNode` stopped after its first page with `GoodNoData`.
>
> **Short pages within a cluster still carry a continuation point.** A within-cluster page that
> returns fewer than `NumValuesPerNode` samples (because the cluster happened to be smaller than
> the cap, or is the final partial batch) is not the last page if the cluster itself has not been
> fully emitted — the server retains the continuation point so the client can drain the remainder.
> Only when the cluster is exhausted and the cursor has advanced past the timestamp does the
> short-page rule apply.
>
> **Cluster larger than `MaxTieClusterOverfetch`.** If the over-fetch itself reaches the ceiling
> without spanning the full cluster, the node fails **loudly** with
> `BadHistoryOperationUnsupported` and the tag + timestamp + ceiling are logged. Remedies: raise
> `MaxTieClusterOverfetch` (or `NumValuesPerNode`) to cover the full cluster, or investigate the
> data anomaly (raw samples normally carry strictly increasing distinct timestamps).
>
> For a single tag's raw history a tie cluster larger than the default 65 536 is a severe data
> anomaly. The ceiling exists to bound server-side memory on pathological data, not to cap normal
> operation.

Continuation points are bound to the OPC UA session (the SDK's
`ServerConfiguration.MaxHistoryContinuationPoints` cap, default 100, with oldest-eviction; points
are disposed when the session closes). Resuming an unknown / evicted / released point returns
`BadContinuationPointInvalid`. `releaseContinuationPoints` drops the stored cursors without reading
data.

### Continuation-point paging (Events) (#669)

A capped event read (`NumValuesPerNode > 0`) pages exactly like Raw, with one difference: the trigger is the **source's
truncation flag** (`HistoricalEventsResult.ContinuationPoint != null`), never the page count, so a cap-sized page the source
says is complete carries no continuation point. `NumValuesPerNode == 0` stays unbounded and never pages.

- The cursor is `(source, next start, end, boundary skip, cap)` in the same session-bound store as Raw (flagged as an events
  cursor — a Raw point presented to an events read, or the reverse, is `BadContinuationPointInvalid`). The next page reads from
  the last event's `EventTimeUtc` **inclusive** and drops the events at that timestamp already emitted.
- A tie cluster larger than the cap is paged within by over-fetching the half-open window `[T, T + resolution)` up to
  `MaxTieClusterOverfetch`; beyond that bound the node answers `BadHistoryOperationUnsupported` (raise the bound).
- **Tie exactness depends on the backend.** SqlServer and QuestDB order by `(EventTimeUtc, EventId)`, so the skip count is exact.
  The Gateway backend has no stable intra-timestamp order (and only a truncation flag), so ties at a page boundary are
  best-effort there.
- Reverse event reads (`start > end`) are not paged server-side; no continuation point is issued for them.
- No session on the request (in-process callers) degrades to single-shot, as for Raw.

### Modified-value history (Part 11)

**Gitea #668.** On the `SqlServer` and `QuestDb` backends every HistoryUpdate leaves a **modification trail**: one row per sample it
inserted, replaced or deleted, carrying the value **as it was before** the change (none for an insert), the caller's principal, the
modification time and the Part 11 `HistoryUpdateType`. A HistoryReadRaw with `IsReadModified = true` returns that trail as a
`HistoryModifiedData` — one `DataValue` (the prior value) plus one parallel `ModificationInfo` (`ModificationTime`, `UpdateType`,
`UserName`) per entry.

| Operation | Recorded `UpdateType` | Prior value |
|---|---|---|
| Insert | `Insert` | none — served `null` / `BadNoData` |
| Replace | `Replace` | the replaced value and its status |
| Update (upsert) | `Update` | the replaced value, or none when it inserted |
| DeleteAtTime / DeleteRaw | `Delete` | the deleted value and its status |

- **Only applied items are recorded.** A refused item (`BadEntryExists`, `BadNoEntryExists`, `BadInvalidTimestamp`, …), a DeleteRaw over an
  empty range, the **live recorder's writes** and the **retention purge** leave nothing.
- **`UserName` is the session's principal** — the API key's `NodeAcl` principal, or `anonymous` for an admitted anonymous session (the
  key id is stored too, not served). Longer than 200 characters is truncated, never refused.
- **Window and order.** Same windows as Raw: forward `[start, end)` ascending by `(SourceTime, ModifiedId)`, reverse (`start > end`)
  `(end, start]` descending, `start == end` the modifications at exactly that time; an unset bound with a cap pages open-ended like Raw.
  Several modifications of one timestamp appear in the order they were made.
- **Paging.** `NumValuesPerNode > 0` pages with session-bound continuation points keyed on the last entry's `(SourceTime, ModifiedId)` —
  an exact, exclusive key, so there are no ties to over-fetch and no trailing empty page (the backend reads `cap + 1` rows to know).
  The cursor is bound to the node and to the modified read kind (a raw or events point is `BadContinuationPointInvalid`).
  `NumValuesPerNode == 0` is unbounded.
- **Authorization** is the ordinary **HistoryRead** decision — no new operation or permission (ungated with `Security:Acl:Enabled` off).
- **Unsupported** ⇒ `BadHistoryOperationUnsupported`: the `Gateway` backend (the AVEVA Historian exposes no modification trail) and the
  `Null` default. **Deleting the trail** (`DeleteRawModifiedDetails.IsDeleteModified = true`) is not supported either — the trail is an
  audit record and is aged out only by retention (below).
- **Atomicity differs by backend.** `SqlServer` writes the trail row with `OUTPUT … INTO` in the **same statement** that changes `hist.Sample`,
  so a change and its trail row commit or roll back together. QuestDB has no multi-table transaction, so the trail row is written **before**
  the change: a crash (or failed sample INSERT) in between leaves an **orphan trail row** describing a change that never landed — never a
  change without its trail row. Treat a QuestDB trail entry whose sample still shows the prior value as such an orphan.
  A failed sample write after its trail rows landed logs a **Warning** naming the tag, the row count and the time range, so the orphan
  can be found.
- **Rolling upgrade (SqlServer).** The v2 schema is applied by the first node that upgrades. Until its peer is upgraded too, that
  not-yet-upgraded node still runs the old build, whose HistoryUpdates write **no trail rows** — silent gaps in the audit trail. Upgrade both
  nodes of a pair together (stop, upgrade, start both); modifications made by an old-build node in the window are never recorded.
- **Paging vs a concurrent HistoryUpdate (SqlServer).** `ModifiedId` is an `IDENTITY`, which orders by allocation, not by commit. A modified
  read paging at the same moment a HistoryUpdate commits a modification of the **same source timestamp** can have already passed that
  row's id and so never see it on this pass (a fresh read does). The cursor is exact for a quiescent trail, not a snapshot guarantee.
- **QuestDB `ModifiedId` across nodes.** Ids are UTC ticks bumped past the last one issued by that process, so a pair's ids interleave by
  each node's **wall clock**. Same-source-time modifications made on two nodes order by their clocks (clock skew can reorder them);
  within one node the order is strict.
- **Retention.** The trail ages out on the same **source-time** cutoff as the samples it describes: `SqlServer` purges `hist.SampleModified`
  per tag alongside `hist.Sample`; QuestDB puts the same TTL on `{p}sample_modified`.

```bash
otopcua-cli historyread -u opc.tcp://localhost:4840 -U apikey -P otua_<keyId>_<secret> \
  -n "ns=2;s=Plant/Modbus/dev1/Speed" --start 2026-01-15T00:00:00Z --end 2026-01-16T00:00:00Z --modified
```

### Total aggregate

The OPC UA `Total` aggregate is **supported** over the HistorianGateway backend. The gateway exposes a
native **`Integral`** retrieval mode, so `Total` maps straight to it (`HistoryAggregateType.Total →
RetrievalMode.Integral`) — no client-side scaling. (This replaces the retired Wonderware path, which had no
`Total` column and derived it client-side as time-weighted `Average × interval-seconds`.) `Count` is
likewise a native gateway mode. Bucket status codes and timestamps are preserved unchanged; empty / null
buckets surface as `BadNoData`.

### Aggregate set and `BadAggregateNotSupported` (#285)

`HistoryAggregateType` carries the wider Part 13 set (Interpolative, TimeAverage, Start, End, Delta, Range,
Min/MaxActualTime, Start/EndBound, DeltaBounds, DurationGood/Bad, PercentGood/Bad, WorstQuality,
NumberOfTransitions, AnnotationCount, StandardDeviationPopulation, VariancePopulation); `OtOpcUaNodeManager.MapAggregate`
maps each standard aggregate NodeId (`TimeAverage2` and `Total2` were appended for #671). The **`SqlServer`** and
**`QuestDb`** backends serve the seventeen listed in the [aggregate table](#aggregates-processed-reads); everything else
(e.g. `StartBound`, `DurationGood`, `StandardDeviationPopulation`) is `BadAggregateNotSupported` for that node. The **HistorianGateway** backend only has retrieval modes for the original
five (Average, Minimum, Maximum, Total, Count): for any other member `AggregateModeMapper` throws
`HistoryAggregateNotSupportedException`, which the HistoryRead path turns into `BadAggregateNotSupported` for that
node only (not a generic failure, and not counted as a backend fault). The OPC UA Client driver forwards every
member upstream. A NodeId outside the mapped set (e.g. `StandardDeviationSample`) is still `BadAggregateNotSupported`.

### Known limitations

- **Processed and AtTime are single-shot** (no continuation points). Unlike Raw, neither
  `ReadProcessedDetails` nor `ReadAtTimeDetails` carries a client count cap (`NumValuesPerNode`):
  the Processed bucket count is deterministic (window / interval) and AtTime returns exactly one
  sample per requested timestamp, so the single-shot backend returns the complete result in one
  read and there is no "full page ⇒ maybe more" signal to page on. Returning the full result with
  no continuation point is spec-conformant.
- **No modified-value history on the `Gateway` backend.** Modified reads (`IsReadModified = true`) are served by
  `SqlServer` and `QuestDb` ([Modified-value history](#modified-value-history-part-11), #668); the HistorianGateway exposes no
  modified/edited-history surface, so on `Gateway` (and with no historian) they return `BadHistoryOperationUnsupported`.

### Redundancy and authorization

History reads are served from any node — there is no Primary gate. The `HistoryRead` AccessLevel bit set at
materialization admits a node to the read dispatch. With `Security:Acl:Enabled=false` (the default) HistoryRead is
**ungated** for any admitted session; with it `true`, the scoped `HistoryRead` grant decides each node
(see [security.md](security.md#data-plane-authorization)). **HistoryUpdate is always gated** — see [HistoryUpdate](#historyupdate).

---

---

## SQL Server backend

Selected with `ServerHistorian:Backend=SqlServer`. Implemented in `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer`
(the project does not reference the Gateway project). All five seams share one connection helper, one schema
bootstrapper and one tag catalog, so the schema is ensured once per process and tag ids are cached once.

### Schema

Created by the embedded idempotent scripts `Schema/001_initial.sql` and `Schema/002_sample_modified.sql` (mirrored at
[`docs/historian/sqlserver/001_initial.sql`](historian/sqlserver/001_initial.sql) and
[`docs/historian/sqlserver/002_sample_modified.sql`](historian/sqlserver/002_sample_modified.sql)); `$(Schema)` is the configured schema.
The bootstrapper applies every script above the recorded version, in order, under the schema applock.

| Object | Purpose |
|---|---|
| `SchemaVersion(Version, AppliedUtc)` | Applied-script ledger. Required version: **2** (since #668). |
| `Tag(TagId int identity, Name nvarchar(400) unique, DataType tinyint, EngineeringUnit, Description, CreatedUtc, UpdatedUtc)` | One row per historian tagname. `DataType` is a `DriverDataType`. |
| `Sample(TagId, SourceTimeUtc datetime2(7), ValueKind tinyint, NumValue float, IntValue bigint, StrValue nvarchar(max), StatusCode int, ServerTimeUtc)` | **PK `(TagId, SourceTimeUtc)`** — one value per timestamp (Part 11), which makes redelivery idempotent and rules out tie clusters. `StatusCode` is the uint status stored bit-for-bit. |
| `Event(EventId nvarchar(64) PK nonclustered, SourceName, AlarmId, AlarmName, AlarmTypeName, Severity, EventKind, Message, UserName, Comment, EventTimeUtc, ReceivedTimeUtc)` | Alarm history. Clustered on `(EventTimeUtc, EventId)`; extra index `(SourceName, EventTimeUtc)`. |
| `SampleModified(TagId, SourceTimeUtc, ModifiedId bigint identity, ModificationTimeUtc, UpdateType tinyint, Principal nvarchar(200), ApiKeyId, ValueKind, NumValue, IntValue, StrValue, StatusCode, ServerTimeUtc)` (v2) | The Part 11 [modification trail](#modified-value-history-part-11): one row per sample a HistoryUpdate changed, carrying the **prior** value columns (all NULL for an insert). Clustered PK `(TagId, SourceTimeUtc, ModifiedId)`. ⚠️ No FOREIGN KEY, CHECK or trigger — it is an `OUTPUT … INTO` target, which forbids them; adding one fails every HistoryUpdate. |
| `SampleRow`, `TimeRow` table types | TVPs for batched sample writes/merges, AtTime and delete-at-time. |

Column mapping per `ValueKind`: `Double` → `NumValue`; `Int64` → `IntValue` + `NumValue` (as a double, for aggregates);
`Boolean` → `IntValue` 0/1 + `NumValue` 0/1; `String` → `StrValue`; `DateTime` → `IntValue` (UTC ticks); `Empty` → all NULL.
`ValueKind` codes: Empty 0, Double 1, Int64 2, Boolean 3, String 4, DateTime 5. Timestamps are bound as `datetime2(7)`
everywhere (full 100 ns precision — never a `DataTable`, which would round to 3.33 ms).

**Applying the scripts by hand** (needed when `AutoCreateSchema=false`): create the database, then run each script in order —
`sqlcmd -S <server> -U <user> -P <pw> -d <db> -I -v Schema=hist -i docs/historian/sqlserver/001_initial.sql`, then the same with
`002_sample_modified.sql` (`-v Schema=` supplies the `$(Schema)` placeholder; verified against the docker-dev SQL container). Re-running is a no-op.

⚠️ **Upgrading an existing v1 historian (#668).** A build with this change requires schema **version 2**. With `AutoCreateSchema=true`
(the default) the first historian call after the upgrade applies `002_sample_modified.sql` in place (it only creates `SampleModified`; existing
samples are untouched). With `AutoCreateSchema=false` **every historian call fails** (`missing version 2 (found 1)`, in health `LastError`)
until an operator runs `002_sample_modified.sql` by hand as above — do that before rolling the new build out. Modifications made before the
upgrade have no trail; the trail starts with the first HistoryUpdate after it.

### Writes

- **Live values** (`IHistorianValueWriter`): resolve `TagId` through a cache, creating the `Tag` row on a miss
  (`MERGE … WITH (HOLDLOCK)`; `DataType` from the first non-Empty value's kind, provisioning corrects it later), then
  `INSERT … SELECT … WHERE NOT EXISTS` through the `SampleRow` TVP. A null timestamp is stamped now;
  `ServerTimeUtc = SYSUTCDATETIME()`. It returns `false` (never throws) on a SQL failure so the recorder's retry/quarantine
  owns it. Semantics: [delivery semantics](#outbox-format-v2-and-rollback).
- **Alarm events** (`IAlarmHistorianWriter`): per event, insert-if-absent keyed by a deterministic `EventId` = lowercase-hex
  SHA-256 over the length-prefixed fields `EquipmentPath, AlarmId, EventKind, TimestampUtc.Ticks, User, Comment, Message`.
  Outcomes: `Ack` on insert or already-present (including a concurrent writer winning a 2627/2601 race); `RetryPlease` on
  transient SQL errors (connection, timeout, deadlock 1205 …), bootstrap failures, cancellation and any unclassified error;
  `PermanentFail` (dead-letter) only for data errors SQL will reject forever (truncation 2628/8152, conversion, null, range, constraint).
  `SourceName` is the event's `EquipmentPath`; `Severity` is the numeric Part 9 value (Low 200, Medium 500, High 700, Critical 900).
- **Tag names** are `nvarchar(400)`, and a longer name is **refused, never truncated** (a truncating parameter would resolve
  it onto whichever tag owns its first 400 characters): `EnsureTags` fails it individually, the live-value writer returns
  `false` for that tag (a per-tag failure, so the #538 quarantine eventually abandons it — logged as a Warning), the updater
  answers `BadInvalidArgument` for every item, and reads treat it as not found (Raw empty, AtTime / Processed `BadNoData`). An
  event read filtered by a source name over 400 characters returns no events. Keep historian tagnames under 400 characters.
- **Schema bootstrap failures are cached for 5 s.** With `AutoCreateSchema=false` and the schema missing (or SQL down), every
  historian call fails fast with the last bootstrap error for 5 s before SQL is tried again, so a missing schema does not
  serialise every write, read and alarm event behind one round-trip each; each retry that fails logs a Warning. If
  `sp_releaseapplock` fails after an auto-create, the connection pool is cleared so the session holding the applock is closed
  rather than returned to the pool.

### Raw reads

`ReadRawAsync` returns every sample as stored — outage markers (`Empty` + Bad status) included. The one exception is defensive:
a row with no value whose stored status is **not** Bad (only a build from before the writer forced `Empty` samples Bad could
have written one) is served with `BadNoData`, on Raw and AtTime alike.

| Request | Window | Order |
|---|---|---|
| `start < end` | `[start, end)` | ascending |
| `end < start` (Part 11 reverse) | `(end, start]` | descending |
| `start == end` | the sample exactly at that time | — |

`NumValuesPerNode == 0` is unbounded (larger values clamp to `int.MaxValue`). An unknown tag is an empty result (`GoodNoData`);
reads **never create** tag rows. The node manager's paging is unchanged; because the PK rules out tie clusters, resuming
"from the last timestamp inclusive and trimming it" is exact.

### AtTime reads

One statement per call: the distinct requested times go in a `TimeRow` TVP and each picks up its previous-or-equal sample `p` and
next sample `n` via two `OUTER APPLY`s. A *usable* sample has severity Good or Uncertain **and** a value (an `Empty` sample is an
outage marker and counts as Bad whatever its code). Per requested time `T` (the result is stamped `T`):

| Case | Result |
|---|---|
| `p` at exactly `T` | the stored value and status, unchanged |
| no `p` (before the first sample), or unknown tag | `BadNoData`, null |
| **`p` is Bad** (`T` lies inside an outage) | **null value carrying `p`'s Bad status code** (info bits stripped) — never better than its source |
| `p` usable, **`n` Bad** | **`p`'s value held** (not interpolated into the outage), `UncertainDataSubNormal` + Interpolated |
| `Double` / `Int64` with a usable `n` | linear interpolation of `NumValue`; `Int64` is computed **through `double`** and rounded to the nearest integer (ties away from zero), keeping the `Int64` type. Status `Good` + Interpolated when both neighbours are Good, else `UncertainDataSubNormal` + Interpolated |
| `Boolean` / `String` / `DateTime`, **or `T` after the last sample** | **stepped**: `p`'s value with `p`'s status + Interpolated |
| `T` after the last sample **and after now** | `p`'s value held, `UncertainDataSubNormal` + Interpolated (the value is not known to still hold) |

Setting Interpolated (or, in aggregates, Calculated) first clears any historian bit the stored status already carries, so a
result never has both.

⚠️ Because `Int64` interpolation goes through `double`, magnitudes beyond 2^53 lose precision.

### Events

`[start, end)` by `EventTimeUtc`; **a reversed window (`end < start`) is swapped and returned ascending**. A null `sourceName` means all
sources, otherwise `SourceName = @source`. Ordered `(EventTimeUtc, EventId)` ascending. A bounded read fetches `maxEvents + 1` rows and
returns a non-null continuation point when the extra row exists; the node manager turns that flag into an OPC UA continuation
point (see [Continuation-point paging (Events)](#continuation-point-paging-events-669)).

### Aggregates (Processed reads)

> **Two kinds.** The *per-sample* aggregates (Average, Minimum, Maximum, Count, Range, Start/End/Delta, Min/MaxActualTime,
> PercentGood/Bad) are simple per-sample definitions — no time-weighting, no interval-boundary values — pushed down to SQL. The
> *time-weighted* aggregates (TimeAverage, TimeAverage2, Total, Total2, Interpolative; #671) follow OPC UA Part 13 with bounding values
> — see [Time-weighted aggregates](#time-weighted-aggregates-part-13-671). Any `HistoryAggregateType` not in the tables throws
> `HistoryAggregateNotSupportedException` ⇒ `BadAggregateNotSupported` for that node (e.g. `StartBound`, `DurationGood`,
> `StandardDeviationPopulation`).
>
> ⚠️ **Release note / migration (#671): `Total` changed meaning on the `SqlServer` and `QuestDb` backends.** It was the plain `SUM` of
> the usable samples in the interval, pushed down to SQL; it is now the Part 13 time-weighted integral, `TimeAverage × interval seconds`
> (value·seconds), **evaluated in C# from the raw rows** (not pushed down) and therefore subject to `ServerHistorian:MaxTimeWeightedRawRows`
> — a long `Total` over a dense tag that used to succeed may now fail `BadTooManyOperations`; read it in shorter windows. This brings
> `Total` closer to the `Gateway` backend, whose `Total` maps to AVEVA's native time-weighted `Integral` retrieval mode — but AVEVA's
> integral time unit/divisor has not been verified, so the two backends may differ by a constant factor for the same data. A client
> that relied on the old sum gets a different number for the same data (e.g. two samples 10 and 20 ten seconds apart no longer total
> 30). There is no switch back; `Count` × `Average` reproduces the old sum.

**Bucketing.** One result per interval of `[start, end)`; bucket *i* = `[start + i·interval, start + (i+1)·interval)`, start-inclusive /
end-exclusive to the tick, the last bucket truncated at `end`. `interval <= 0` ⇒ a single interval `[start, end)`; `start == end` ⇒ no
intervals. The bucket index is computed in **100 ns ticks** (not milliseconds): `DATEDIFF_BIG(microsecond, @start, t) × 10` plus a
sub-microsecond correction from the two tick digits, divided by the interval in ticks. (`DATEDIFF` counts unit *boundaries crossed*, so a
millisecond formulation misplaces samples near a bucket edge, and a nanosecond one overflows past ≈292 years.) A window of more than
**100 000 intervals** is refused before any I/O and reported `BadHistoryOperationUnsupported` for that node. **Reverse windows**
(`end < start`) are swapped, computed forward **anchored at the earlier time**, and returned in **descending** bucket order. Every returned
status carries the historian *Calculated* info bit, `BadNoData` included.

**Sample classes.** *Usable* = severity Good or Uncertain **and** a stored value (not `Empty`). *Good* = usable with severity Good.
*Bad* = not usable.

| Aggregate | Computed over | Value timestamp |
|---|---|---|
| `Average`, `Minimum`, `Maximum` | `AVG` / `MIN` / `MAX` of `NumValue` over usable samples | bucket start |
| `Count` | number of usable samples (`Int32`) | bucket start |
| `Range` | `Maximum − Minimum` | bucket start |
| `Start`, `End` | first / last usable sample by time (keeps the sample's type) | bucket start |
| `Delta` | `End − Start` (numeric samples) | bucket start |
| `MinimumActualTime`, `MaximumActualTime` | the usable sample with the smallest / largest `NumValue`; ties ⇒ earliest | **the sample's own time** |
| `PercentGood` | `100 × (Good samples) / (all samples)` | bucket start |
| `PercentBad` | `100 × (Bad samples) / (all samples)` | bucket start |

Status and edge rules:

- An **empty interval** (no samples) ⇒ `BadNoData`. **`Count` — and every value aggregate — over an interval whose samples are all Bad ⇒ `BadNoData`**, not `0`.
- Value aggregates are `Good` when every sample in the bucket is Good and `UncertainDataSubNormal` when the bucket also holds any Bad **or Uncertain** sample.
- `Count`, `Start`, `End`, `PercentGood`, `PercentBad` accept any value kind; the other aggregates (and `Delta`) on a bucket that holds a usable `String`/`DateTime` sample ⇒ `BadAggregateInvalidInputs` for that interval.
- **`PercentGood` / `PercentBad` and Uncertain samples:** Uncertain samples are counted in the denominator but are **neither Good nor Bad**, so `PercentGood + PercentBad < 100` whenever a bucket holds Uncertain samples. Both report status `Good` for any non-empty bucket.
- Everything except `Count` and `Start`/`End` is a `double`; `Int64` samples aggregate as doubles and `Boolean` as 0/1.

### Time-weighted aggregates (Part 13, #671)

`TimeAverage`, `TimeAverage2`, `Total`, `Total2` and `Interpolative`, evaluated by the shared
`Driver.Historian.Sql.Common/TimeWeightedAggregator.cs` for **both** SQL backends (so `SqlServer` and `QuestDb` agree exactly).
**This whole family — `Total` included — is evaluated in C#, not pushed down to SQL** (see *Pushdown trade-off* and the raw-row cap
below). Bucketing (edges, truncated last bucket, the 100 000-interval cap) is the same as above; reverse windows are covered below.

**Server defaults, fixed:** `TreatUncertainAsBad = false`, `PercentDataBad = PercentDataGood = 100`, `UseSlopedExtrapolation = false`.
**A client's `AggregateConfiguration` in the HistoryRead request is ignored** (the node manager does not forward it); every read uses
these defaults.

**Bounding values.** Besides the window `[start, end)`, each read fetches four single-row seeks: the latest *usable* sample before
`start`, the latest sample of any quality before `start` (they differ when an outage straddles the start), the first sample of any
quality at or after `end`, and the first *usable* one at or after `end`. So a bucket with no samples of its own, between two samples,
still gets a value.

**The signal.** Between a usable sample `p` and the next sample `s`: both `Double`/`Int64` ⇒ **sloped** (linear), segment quality the
worse of the two; either `Boolean` ⇒ **stepped** (`p` held), quality `p`'s; `s` Bad ⇒ `p` held up to the outage (never interpolated into
it — the AtTime rule), Uncertain; no later sample at all ⇒ `p` held to the window end (stepped extrapolation), Uncertain. From a Bad
sample (incl. an `Empty` outage marker) to the next sample the signal is **Bad** (outage); before the first sample there is **no
data**. That is the
*simple-bounding* signal. The *interpolated-bounding* signal skips Bad samples: it joins consecutive usable samples (sloped / stepped)
straight across an outage.

| Aggregate | Value | Bad intervals |
|---|---|---|
| `TimeAverage` | ∫ interpolated-bounding signal ÷ its defined duration | interpolated across (Part 13) |
| `Total` | `TimeAverage × interval seconds` (value·s; the truncated last bucket uses its own length) | interpolated across |
| `TimeAverage2` | ∫ simple-bounding signal over non-Bad time ÷ non-Bad duration | excluded |
| `Total2` | ∫ simple-bounding signal over non-Bad time (value·s) | excluded |
| `Interpolative` | interpolated-bounding value **at the interval start** (reverse read: see below) | interpolated across |

**Status, the four time-weighted values** — aligned with the OPC Foundation SDK's `AggregateCalculator.GetTimeBasedStatusCode`
(inspected in `Opc.Ua.Server` 1.5.378.106) under the fixed defaults. With `TreatUncertainAsBad = false`, **Uncertain time counts as
Good time**: Uncertain samples, a value held up to an outage and the stepped extrapolation past the last sample all count toward
PercentDataGood.

- Every instant of the interval non-Bad ⇒ `Good`.
- Some non-Bad time but not all (an outage, or time before the first sample) ⇒ `UncertainDataSubNormal`.
- No non-Bad time but some outage time ⇒ generic **`Bad`** (`0x80000000`) — the SDK's all-Bad result, value null.
- No data at all ⇒ `BadNoData`.
- A usable `String`/`DateTime` sample feeding the interval ⇒ `BadAggregateInvalidInputs`.

All carry the Calculated bit; a non-Bad result for the **truncated last interval** (shorter than the processing interval) also carries
the **Partial** bit (`0x4`, `AggregateBits.Partial`). Timestamp = interval start; value `double` (Boolean 0/1). (The per-sample
aggregates do not set Partial — a follow-up.)

> **A live tag's latest interval.** Past the tag's last stored sample the value is held (stepped extrapolation, Uncertain). Under
> the SDK rule that Uncertain time is Good time, so the latest interval of a live tag normally reads **`Good`** — not Uncertain —
> with the last value held to the interval end. It reads Uncertain only if the interval also contains outage time or starts
> before the tag's first sample. Do not take a Good latest interval as proof that data arrived up to its end. If it was truncated
> by the request's end time it also carries **Partial**.

**Status, `Interpolative`** (point-based, not the PercentDataGood rule):
- A usable sample exactly at the point ⇒ that value with **its own stored status** (no historian bit).
- No usable sample at or before the point ⇒ plain `BadNoData`.
- Otherwise the value with the **Interpolated** bit:
  - `Good` on a segment whose defining samples are Good;
  - `UncertainDataSubNormal` inside or up to an outage (bridged) and past the last sample (extrapolated).

**Reverse reads (`end < start`).** Part 13 anchors a reverse read at the request's start, which is the later time. `Interpolative`
follows that: its points are `start, start − interval, …` (in the forward window's terms `End − i·interval`), latest first. A
reverse read 10:00 → 09:00 at 1 h returns the value at **10:00**, not 09:00, and no point ever falls before the window. The interval
aggregates (TimeAverage, Total, …) keep the existing backend-wide reverse rule: computed forward with intervals anchored at the
earlier time, returned in descending order. Re-anchoring those at the later time for every aggregate is a follow-up.

**Pushdown trade-off.** Unlike the per-sample aggregates, the time-weighted family is **not** pushed down to SQL: the backend streams
the window's rows plus the four bounds (one batch of five result sets on SQL Server; a bounds `UNION ALL` + a window select on QuestDB)
and the evaluator splits segments at bucket edges in C#. Network traffic is therefore O(samples in the window) rather than O(buckets);
evaluation memory stays O(buckets) (rows are streamed, never materialised). The alternative — the same Part 13 segment
interpolation, edge splitting and two bounding definitions written twice, in T-SQL and QuestDB SQL, and kept bit-identical — was
judged the larger risk. For very long, densely sampled windows prefer the pushed-down `Average` (or a coarser read) over
`TimeAverage`.

**Raw-row cap.** The window query is limited to `ServerHistorian:MaxTimeWeightedRawRows` + 1 rows (default 1 000 000, shared by both
SQL backends; `TOP (@maxRows)` / `LIMIT $4`). When that extra row arrives the read stops and the node fails with
**`BadTooManyOperations`** — for that node only; it is not counted in the backend's health failures. Without the cap a one-year
`Total` on a 1 Hz tag (≈31 M rows) would hold a pooled connection until `HistoryReadDeadline` and then return `BadTimeout`. The four
bound rows do not count toward the cap, and the per-sample aggregates are never capped. Raise the cap, or read in shorter windows.

### Health

The read client reports `GetHealthSnapshot()` (query/failure counters, consecutive failures, `LastError` — e.g. a missing schema under
`AutoCreateSchema=false`). A SQL error on a read surfaces as `BadHistoryOperationUnsupported` for **that node only**; a cancelled read
(deadline, client abort) is not counted as a backend fault.

### Retention (`RetentionDays`)

`hist.Sample`, `hist.SampleModified` and `hist.Event` grow without bound unless `ServerHistorian:SqlServer:RetentionDays` is set. QuestDB already has an
equivalent TTL (`QuestDb:RetentionDays`); the Gateway backend's retention is the AVEVA Historian's own.

- **What runs.** `HistorianRetentionService` (a hosted service, driver-role nodes, Backend=`SqlServer` with `RetentionDays` set; inert
  otherwise) ticks every `PurgeInterval` (first pass at start-up) and calls the backend's `IHistorianRetention.PurgeAsync`.
- **Gate vs. mutual exclusion.** Each tick consults `IHistorizationRoleView.ShouldRecordHistorizedValues` (the capture verdict; a
  standby skips, an unknown role runs). That verdict is **pair-scoped** while the purge is **database-scoped**, so the gate is only a
  load reducer: the `sp_getapplock` below is the actual mutual exclusion. Pairs sharing one historian database, and both nodes of a
  pair on their boot-time pass, simply serialise on it. Set `RetentionDays` **uniformly on every node that shares a historian
  database** — if they differ, the smallest window wins.
- **How it deletes.** Cutoff = `UtcNow - RetentionDays` on source time. Samples are purged **per tag** with
  `DELETE TOP (@n) ... WHERE TagId = @id AND SourceTimeUtc < @cutoff` (a seek on the clustered `(TagId, SourceTimeUtc)` key; a global
  time predicate would scan) — and, in the same per-tag pass, the modification trail with
  `DELETE TOP (@n) FROM SampleModified WHERE TagId = @id AND SourceTimeUtc < @cutoff` (#668; same **source-time** cutoff, clustered seek; the
  purge itself never writes a trail row) — then events with `DELETE TOP (@n) ... WHERE EventTimeUtc < @cutoff` (seek on `CX_Event_Time`), each looped
  until a batch comes back short. Cancellation is honoured between batches. The `hist.Tag` catalogue is never purged.
- **Shared databases.** Several pairs may point at one database; a session `sp_getapplock` (`OtOpcUa.Historian.Retention.<schema>`,
  timeout 0) lets one purger run and the others skip that tick (Debug log).
- **Observability.** Counter `otopcua.historian.retention.purged{table=Sample|SampleModified|Event}`; a failed pass logs a Warning and retries next
  interval — it never affects capture or reads.
- **Not done: partition switching.** For very large installs (billions of rows) a partitioned `hist.Sample` with sliding-window
  `SWITCH`/drop is cheaper than row deletes; the purger is deliberately simple batched deletes. Another table is added by one more
  `DeleteLoopAsync` call in `SqlServerHistorianRetention.PurgeAsync` (as `hist.SampleModified` was, #668).

### Docker-dev recipe

The docker-dev `sql` container (`otopcua-dev-sql-1`, host port `14330`, `sa` with the committed dev password from
`docker-dev/docker-compose.yml`) already hosts the ConfigDb; give the historian its own database on the same instance.

```bash
# 1. Create the database (the backend creates the schema + tables on first use, not the database).
sqlcmd -S localhost,14330 -U sa -P 'OtOpcUa!Dev123' -C -I -Q "CREATE DATABASE OtOpcUaHistorian"

# 2. Point the central nodes at it. Add to the `environment:` of central-1 AND central-2 in docker-dev/docker-compose.yml
#    (or a compose override), rebuild the images (the rig image goes stale), then `up -d`:
#      ServerHistorian__Enabled: "true"
#      ServerHistorian__Backend: "SqlServer"
#      ServerHistorian__SqlServer__ConnectionString: "Server=sql,1433;Database=OtOpcUaHistorian;User Id=sa;Password=OtOpcUa!Dev123;TrustServerCertificate=True"
#      ContinuousHistorization__Enabled: "true"
#      ContinuousHistorization__OutboxPath: "/app/data/historization-outbox"
#    (ServerHistorian__Endpoint / __ApiKey are not used by this backend.)

# 3. Mark a rig tag historized ("isHistorized": true in its TagConfig), deploy, then watch rows land:
sqlcmd -S localhost,14330 -U sa -P 'OtOpcUa!Dev123' -C -I -d OtOpcUaHistorian \
  -Q "SELECT t.Name, COUNT(*) n, MAX(s.SourceTimeUtc) last FROM hist.Sample s JOIN hist.Tag t ON t.TagId=s.TagId GROUP BY t.Name"
```

Only the Primary of a pair records, so the row count advances once per published value, not twice. To watch retention, add
`ServerHistorian__SqlServer__RetentionDays: "1"` and `ServerHistorian__SqlServer__PurgeInterval: "00:00:30"`, insert a `hist.Sample` row
with an old `SourceTimeUtc` for an existing tag, and see it vanish within a tick while recent rows stay. Read it back with
`Client.CLI historyread` and rewrite it with `Client.CLI historyupdate` ([Client.CLI.md](Client.CLI.md)). The password above is the
committed docker-dev credential, not a secret; use `${secret:…}` / environment for anything real.

### Integration tests

`tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests` (bootstrap incl. two concurrent bootstrappers, every kind
round-trip, redelivery, raw / AtTime / aggregate / event reads, every update result code) is gated on the environment variable
**`HISTORIAN_SQL_TEST_CONNECTION`** — a connection string with `CREATE`/`DROP DATABASE` rights. Absent ⇒ every test skips. Present ⇒ each
run creates a throwaway `OtOpcUaHistorian_T_<guid>` database and drops it on dispose.

```bash
export HISTORIAN_SQL_TEST_CONNECTION='Server=localhost,14330;User Id=sa;Password=OtOpcUa!Dev123;TrustServerCertificate=True;Encrypt=False'
dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests
```

---

## QuestDB backend

Selected with `ServerHistorian:Backend=QuestDb`. Implemented in `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb`
(pure logic shared with the SQL Server backend lives in `Driver.Historian.Sql.Common`). All access is the **Postgres wire protocol via
`Npgsql`** (no ILP, no extra package). Design: [plans/2026-10-10-questdb-historian-design.md](plans/2026-10-10-questdb-historian-design.md);
the spike that pinned every QuestDB-specific form: [plans/2026-10-10-questdb-spike.md](plans/2026-10-10-questdb-spike.md). Pinned image:
`questdb/questdb:10.0.1`.

QuestDB serves every value kind (`Double`, `Int64`, `Boolean`, `String`, `DateTime`, `Empty`), stores the real status, records comms-loss
outage markers, serves the seventeen aggregates (time-weighted ones through the same shared evaluator, so results are identical) and supports **HistoryUpdate** (soft delete), exactly like `SqlServer`. The same conformance
suite (`tests/Drivers/Historian.Conformance`) runs against both backends; the differences below are the only ones.

### Configuration (`ServerHistorian:QuestDb`)

Read only when `Backend=QuestDb`; bound by `HistorianBackendRegistry.BindQuestDb`, validated (consumer-gated, like `SqlServer`) by
`QuestDbHistorianOptions.Validate()`. The gateway keys (`Endpoint`, `ApiKey`, `UseTls`) are not required.

| Key | Default | Validation | Meaning |
|---|---|---|---|
| `QuestDb:ConnectionString` | `""` | **Required** (non-blank) | Npgsql connection string for QuestDB's PG-wire port, e.g. `Host=questdb;Port=8812;Username=admin;Password=quest;Database=qdb;ServerCompatibilityMode=NoTypeLoading`. `ServerCompatibilityMode=NoTypeLoading` skips the `pg_type` load; every query form was verified with it. Supply via env `ServerHistorian__QuestDb__ConnectionString` / `${secret:}` for anything real. |
| `QuestDb:TablePrefix` | `hist_` | `^[A-Za-z0-9_]+$` | QuestDB has no schemas, so the four tables are `{prefix}sample`, `{prefix}event`, `{prefix}tag` and `{prefix}sample_modified` (#668). Spliced into DDL/DML, hence the regex. |
| `QuestDb:RetentionDays` | unset | `>= 1` when set | Applied as table TTL to `{prefix}sample`, `{prefix}event` and `{prefix}sample_modified` (see [Retention](#retention-ttl)). Unset keeps data forever. |
| `QuestDb:CommandTimeout` | `00:00:30` | `> 0` | Per-command timeout, rounded up to whole seconds (minimum 1 s). |
| `QuestDb:AutoCreateSchema` | `true` | — | `true`: create the tables (and reconcile the TTL) on first use. `false`: only verify the four tables exist (⚠️ an install from before #668 lacks `{prefix}sample_modified` — create it with the DDL in `QuestDbTables.CreateSampleModified` before upgrading, or every call fails); if one is missing every historian call fails with `HistorianSchemaMissingException` (health `LastError`, no crash) and no TTL is touched. A failed bootstrap is not latched: it is re-thrown for 5 s without touching QuestDB, then retried. |
| `QuestDb:VisibilityTimeout` | `00:00:05` | `>= 0` | How long a write waits for its rows to become visible to queries ([WAL visibility](#wal-visibility-and-the-read-your-writes-wait)). `0` disables the wait. |

⚠️ **Connection pool size.** QuestDB's default limit is **64** concurrent PG-wire connections (`pg.net.connection.limit`), while Npgsql's
own default `Maximum Pool Size` is **100**. A busy node (HistoryRead batches, the recorder, the updater and alarm writes all share one pool)
could therefore be refused by QuestDB before Npgsql's pool is full. **When the connection string does not set `Maximum Pool Size`
(or `MaxPoolSize`), the backend now caps it at 24**, so the two nodes of a redundant pair sharing one QuestDB stay under the server limit
together (2 × 24 = 48). An explicit value in the connection string is always kept as written — including one above 64, since QuestDB's
limit is server-configurable. The effective value is logged once at Information when the backend first builds its pool.

### Tables

All are **WAL tables**; the first three carry **`DEDUP UPSERT KEYS`**, so re-sending a row with the same key replaces it (the upsert is what
makes HistoryUpdate Replace work; the writer's own insert-if-absent semantics are built on top, below). The modification trail is append-only.

| Table | Key | Partitioning |
|---|---|---|
| `{p}sample` (`ts, tag, kind, num_value, int_value, str_value, status, deleted, server_ts`) | `(ts, tag)` | `PARTITION BY DAY` (TTL when configured) |
| `{p}event` (one-for-one with `hist.Event`) | `(ts, event_id)` | `PARTITION BY DAY` (TTL when configured) |
| `{p}tag` (`ts, name, data_type, eng_unit, description, updated_utc`) | `(ts, name)` with `ts` fixed at `1970-01-01` — an upsert by name | `PARTITION BY YEAR`, never expires |
| `{p}sample_modified` (`ts` = the sample's source time, `tag, modified_id, modification_ts, update_type, principal, api_key_id, kind, num_value, int_value, str_value, status, server_ts`) (#668) | none — append-only, no DEDUP; `modified_id` (process-monotonic ≈ UTC ticks) orders a timestamp's trail | `PARTITION BY DAY` (TTL when configured). A NULL `status` marks "no prior value" (QuestDB `BYTE` has no NULL, so `kind` is 0) |

`status` holds the uint OPC UA status code **unsigned** in a `LONG` (so "Bad" is `status >= 2147483648`, not `< 0`); `int_value` also holds
Booleans as 0/1; `kind` is the `HistorianValueKind` in a `BYTE`. Tag names are limited to 400 characters (longer is a per-tag failure,
never truncated). `{p}tag` is the **only** "is this tag known" authority (`QuestDbTagCatalog`): reads never create a tag, Insert/Update and the
live writer do, Replace and the deletes do not.

### Timestamps: microseconds

QuestDB's designated timestamp is a microsecond. Every inbound `DateTime` is normalised by `QuestDbConnection.Round` (Unspecified ⇒ UTC, Local
⇒ UTC, ticks truncated to a multiple of 10, `MinValue`/`MaxValue` nudged inward because Npgsql would send them as ±infinity) and bound only
through `BindTimestamp`. Consequences:

- Two live samples for one tag **less than 1 µs apart** collapse to one; **the first wins**. A batch is deduplicated client-side after rounding.
- **Raw and Processed results carry stored (µs) timestamps.** A sub-µs request bound is rounded, and the rounded start *is* the request start,
  so a Processed bucket's timestamp is the rounded window start (a sub-µs label divergence from `SqlServer`, which labels with the exact
  start; values and bucket membership are identical).
- **AtTime** evaluates at the rounded time but **stamps each result with the time the client asked for**, so a client can match answers to
  requests. Stored data is µs-only, so a sub-µs request adjacent to a sample is an exact hit.
- `IHistorianDataSource.TimestampResolutionTicks` is **1** by default and **10** on QuestDB. HistoryRead-Raw paging steps off a drained
  timestamp by that amount (forward `T + resolution`, reverse `T − resolution`). Stepping by 1 tick would be truncated back to `T` and a
  one-value-per-page forward read would loop forever. Same-timestamp ties (legal on the Gateway backend) are still paged within the timestamp.
- **Pre-1970 timestamps cannot be stored** (QuestDB: `designated timestamp before 1970-01-01 is not allowed`; the whole INSERT fails). So:
  - the **live writer** drops such samples with at most one Warning per tag per 10 minutes (a device with an unset clock would otherwise warn every drain pass; every dropped sample is still counted in `DroppedPre1970`) — a retry could never succeed, and sending
    them would push the tag into the #538 quarantine;
  - **HistoryUpdate** Insert / Replace / Update of such an item returns `BadOutOfRange` (nothing is sent); `DeleteAtTime` returns
    `BadNoEntryExists`; `DeleteRaw` finds nothing;
  - an alarm event with such a time is a `PermanentFail` (dead-letter);
  - **read bounds before 1970 work** (a Raw/Processed window may start before the epoch; bounds are never clamped, only `MinValue`/`MaxValue` nudged).
- **`Int64` `long.MinValue`.** QuestDB's `LONG` NULL is `long.MinValue`, so a stored `FromInt64(long.MinValue)` has `int_value` NULL. The read path
  repairs it (`kind = Int64 ∧ int_value IS NULL ∧ num_value = −2^63` ⇒ `long.MinValue`); without the repair it would read back as `Empty`/0. Likewise
  `NaN`/`±Infinity` doubles read back as NULL, so they are stored as `Empty` + `BadOutOfRange` (live) or rejected `BadInvalidArgument` (HistoryUpdate), as on `SqlServer`.

### Live writes: insert-if-absent, tombstones included

The recorder's writer (`QuestDbHistorianValueWriter`) has the same first-wins redelivery contract as SQL Server:

1. group by tag, round to µs, drop in-batch duplicates (first wins), drop pre-1970 samples;
2. read the timestamps that already exist for the tag (one range query per cluster of nearby times, **`deleted` rows included**);
3. `INSERT` only the absent rows — parameterised multi-row, at most **1,000 rows per statement**;
4. [wait for WAL visibility](#wal-visibility-and-the-read-your-writes-wait).

Because `DEDUP UPSERT` would otherwise let a re-sent sample overwrite the stored one, step 2 is what keeps outbox redelivery idempotent — and
**a live-writer redelivery never resurrects a deleted row**: a tombstone counts as present. Any SQL or transport failure returns `false`
(the recorder retries; the #538 per-tag quarantine still applies); the writer never throws.

### WAL visibility and the read-your-writes wait

QuestDB applies WAL commits asynchronously, so a row just inserted may not be visible to the next query (measured: up to 7 of 20 inserts invisible
to an immediate re-read). **Every write path waits** — the live writer, the HistoryUpdate updater (including between DeleteRaw pages), the tag
catalog/provisioner and the alarm writer (`QuestDbWalVisibility`): after the INSERT(s) it reads `sequencerTxn` from `wal_tables()` and polls
`writerTxn` about every 10 ms until it catches up, the table is reported `suspended` (Warning, stop), or `VisibilityTimeout` elapses (Warning).
A timeout never changes a result code and never discards data; the rows simply become visible shortly after. Measured: median 0 ms, p95 6 ms.

### HistoryUpdate (soft delete)

All five operations are supported, and **every one is an `INSERT`**: QuestDB `UPDATE`/`DELETE` are never issued. "Exists" means a row that is
present **and not deleted**.

| Operation | Effect | Result |
|---|---|---|
| Insert | write the row when none exists — **a tombstone counts as absent, so Insert over a tombstone succeeds** | `GoodEntryInserted` / `BadEntryExists` |
| Replace | overwrite (dedup upsert) when a live row exists | `GoodEntryReplaced` / `BadNoEntryExists` |
| Update | write either way | `GoodEntryReplaced` or `GoodEntryInserted` |
| DeleteAtTime | write a tombstone (`deleted = true`, value NULL, `BadNoData`) where a live row exists | `Good` / `BadNoEntryExists` |
| DeleteRaw | tombstone every live row in `[start, end)`, 1,000 per page, waiting for visibility between pages | the same codes as `SqlServer` |

**Modification trail (#668).** Each operation first appends one `{p}sample_modified` row per sample it will change, carrying the live row's
current value (read by the same existence query), then writes the change. No multi-table transaction exists, so this order is the guarantee:
a crash between the two leaves an **orphan trail row**, never an untracked change — the documented deviation from `SqlServer`'s atomic trail.
See [Modified-value history](#modified-value-history-part-11).

The per-item codes come from the shared `HistoryUpdatePlanner`, so they equal `SqlServer`'s (including the duplicate-timestamp-in-one-request rules
above). **Every read and aggregate filters `NOT deleted`.** A client Insert at a deleted timestamp re-populates it (the OPC UA intent); only the
*live writer's* redelivery is barred from doing so.

⚠️ **Concurrency.** The updater serializes calls **per tag, in-process only** (64 striped locks held from the existence read through the visibility
wait), so two concurrent Insert/Replace/Delete batches on one tag cannot both act on a stale read. It does **not** lock against the live writer,
nor against a second node. The existence checks read committed data, so a recorder sample at the same timestamp still in flight in the WAL can race
an Insert/Replace (updates normally target past timestamps), and two nodes writing the same *new* key concurrently can produce last-write-wins
(QuestDB has no `UPDLOCK`; the Primary gate makes it rare). This residual is accepted.

### Reads

Every query filters `NOT deleted`; every value goes through `StoredSample`, so the quality clamp equals `SqlServer`'s.

- **Raw:** `ts` range, `ORDER BY ts [DESC] LIMIT $n` (bound, `int`); `ContinuationPoint` is always null — the node manager pages by re-reading
  from the last timestamp ± `TimestampResolutionTicks`.
- **AtTime:** previous-or-equal and next neighbour of each time by `LIMIT 1` seeks, `UNION ALL`-ed, **50 times per statement** (QuestDB 10.0.1 refuses
  more than 126 branches). `AtTimeEvaluator` decides each value, as on `SqlServer`.
- **Events:** `[start, end)` by `ts`, ordered `(ts, event_id)`, `maxEvents + 1` fetched to produce the continuation point.
- **Processed:** one query per node; bucket index `(cast(ts AS LONG) * 10 - startUnixTicks) / intervalTicks` (never `SAMPLE BY`, whose alignment
  differs), so bucket edges are identical to `SqlServer` (start-inclusive, end-exclusive, partial last bucket, interval 0 ⇒ one bucket, 100,000-bucket cap).
  Min/max times use `row_number()` windows with the earliest-timestamp tie-break.

### Retention (TTL)

`RetentionDays` becomes `TTL n DAYS` on `{p}sample`, `{p}event` and `{p}sample_modified` (never on `{p}tag`: TTL must be a multiple of the partition size). QuestDB drops
expired day partitions itself, tombstones included.

- At bootstrap the current TTL is read from `tables()` (`ttlValue`, `ttlUnit`). QuestDB **normalises the unit** (7 days reads `1 WEEK`), so the
  bootstrapper compares in **hours** (`HOUR`=1, `DAY`=24, `WEEK`=168; any other unit counts as different) and issues
  `ALTER TABLE … SET TTL n DAYS` only on a difference (`SET TTL 0 DAYS` removes it). The `ALTER` is applied through the WAL, so it does not
  re-read to verify; a very large `RetentionDays` that QuestDB normalises into MONTH/YEAR is re-`ALTER`ed on every boot (harmless).
- ⚠️ **TTL is enforced only on write commits.** QuestDB checks for expired partitions when a write commits to the table, so **a quiet table never
  drops anything**: partitions older than the TTL linger until the next write lands. Do not read a stale-looking partition as a broken TTL.
- Expiry is judged against the wall clock, so a future-dated sample does not expire current data.
- The TTL is visible in table metadata: `SELECT table_name, ttlValue, ttlUnit FROM tables() WHERE starts_with(table_name, 'hist_')`.

### Docker-dev recipe

QuestDB is an **optional compose overlay**, `docker-dev/docker-compose.questdb.yml`, so the default rig is unchanged. See
[`docker-dev/README.md`](../docker-dev/README.md#questdb-historian-overlay).

```bash
# 1. Bring up ONLY the questdb service (PG wire on host :18812, console/REST on host :19000).
docker compose -f docker-dev/docker-compose.yml -f docker-dev/docker-compose.questdb.yml up -d --no-deps questdb

# 2. Recreate the central nodes against it (the QuestDb connection string already defaults to the `questdb` service):
OTOPCUA_HISTORIAN_ENABLED=true OTOPCUA_HISTORIAN_BACKEND=QuestDb OTOPCUA_CONTHIST_ENABLED=true \
  docker compose -f docker-dev/docker-compose.yml -f docker-dev/docker-compose.questdb.yml up -d central-1 central-2

# 3. Mark a rig tag historized, deploy, then watch rows land (REST, no client needed):
curl -G localhost:19000/exec --data-urlencode "query=SELECT tag, count() n, max(ts) last FROM hist_sample WHERE NOT deleted GROUP BY tag"
```

Only the Primary of a pair records. Read back with `Client.CLI historyread`, rewrite with `Client.CLI historyupdate`.

### Integration tests

`tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests` (bootstrap incl. two concurrent bootstrappers, TTL via `tables()`, writes,
tombstones and redelivery, the visibility wait, updater, reads, paging, and the shared conformance suite) is gated on
**`HISTORIAN_QUESTDB_TEST_CONNECTION`** — an Npgsql connection string for the PG-wire port. Absent ⇒ every test skips. Each test class uses a
unique throwaway prefix (`t_<guid>_`) and drops its tables afterwards. The QuestDB server allows 64 connections, so the fixture prunes idle pool connections.

```bash
docker compose -f docker-dev/docker-compose.yml -f docker-dev/docker-compose.questdb.yml up -d --no-deps questdb
export HISTORIAN_QUESTDB_TEST_CONNECTION='Host=localhost;Port=18812;Username=admin;Password=quest;Database=qdb;ServerCompatibilityMode=NoTypeLoading'
dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests
```

---

## HistoryUpdate

OPC UA **HistoryUpdate** lets a client insert, replace, update and delete stored raw history. It is available only when the selected
backend provides an updater — **`SqlServer` and `QuestDb` do; `Gateway` does not**, and an updater is registered only when `ServerHistorian:Enabled=true`.
Updates go straight to the backend (not through the outbox), are **not Primary-gated** (the database is shared), and share the HistoryRead
batch limiter and `HistoryReadDeadline`.

### Operations

| OPC UA request | Server override | Per-item results (`OperationResults`) |
|---|---|---|
| `UpdateDataDetails`, `PerformInsertReplace = Insert` | `HistoryUpdateData` | `GoodEntryInserted`; `BadEntryExists` if a sample already exists at that timestamp |
| …`Replace` | `HistoryUpdateData` | `GoodEntryReplaced`; `BadNoEntryExists` if none exists |
| …`Update` | `HistoryUpdateData` | `GoodEntryInserted` or `GoodEntryReplaced` (upsert) |
| `DeleteRawModifiedDetails` (`IsDeleteModified=false`) | `HistoryDeleteRawModified` | one node-level code: `Good`; `BadNoData` if `[start, end)` held nothing |
| `DeleteAtTimeDetails` | `HistoryDeleteAtTime` | per time: `Good`; `BadNoEntryExists` |

Each backend call is applied as a unit (SqlServer: one transaction, `MERGE … OUTPUT $action` against the TVP; QuestDb: an INSERT per
chunk plus the WAL-visibility wait, no transaction — see [QuestDB backend](#questdb-backend)). Every applied item is also recorded, with the
caller's principal, in the [modification trail](#modified-value-history-part-11) (#668). The node-level `StatusCode` stays `Good` when item results are
mixed (Part 11 reports item outcomes in `OperationResults`); it is Bad only for whole-node failures.

**Per-item rejections** (answered without touching SQL; other items still proceed): a null item ⇒ `BadInvalidArgument`;
`SourceTimestamp == DateTime.MinValue` ⇒ `BadInvalidTimestamp`; a value the historian cannot type (array, unsupported CLR type) ⇒
`BadTypeMismatch`; a **`NaN` / `±Infinity` `Double` ⇒ `BadInvalidArgument`** — it cannot live in a SQL `float`, and unlike the live-value
writer (which stores it as `Empty`) an operator's explicit update is rejected rather than silently rewritten; and a **null value with a
Good or Uncertain status ⇒ `BadInvalidArgument`** — an `Empty` sample is always Bad, so a null value is accepted only with a Bad status
(an operator-authored outage marker). A caller-supplied status code is otherwise stored as given.

**No DataType check.** HistoryUpdate does **not** compare a value's type with the node's `DataType`: a value is stored as sent (a `String`
written to a `Double` node's history is stored and later read back as a `String`), exactly as the live recorder stores whatever type the
driver publishes. Only a value the historian cannot represent at all is refused (`BadTypeMismatch`, above). Part 11 §6.8 permits a
server to answer `BadTypeMismatch` for a value of the wrong type; this server deliberately does not.

**`DeleteRawModifiedDetails` needs both bounds:** a `StartTime` or `EndTime` of `DateTime.MinValue` (the wire's "unspecified") is
`BadInvalidArgument` for that node and never reaches the backend.

**Duplicate timestamps inside one request** (compared at 100 ns): *Insert* — the first occurrence is applied, later ones are `BadEntryExists`;
*Replace* / *Update* — the last one wins and every earlier occurrence reports the winner's code; *DeleteAtTime* — the first reports the real
outcome, later ones `BadNoEntryExists`. **Unknown tag:** Insert / Update create the tag row; Replace and the deletes report `BadNoEntryExists`
(DeleteRaw: `BadNoData`) without creating anything.

**Unsupported ⇒ `BadHistoryOperationUnsupported`** (per node): the node is not historized; the backend has no updater (`Gateway`, or
`ServerHistorian:Enabled=false`); `PerformUpdateType.Remove`; `DeleteRawModifiedDetails.IsDeleteModified=true` (the
[modification trail](#modified-value-history-part-11) is an audit record, aged out only by retention); and the out-of-scope detail types — **event** HistoryUpdate (`UpdateEventDetails` / `DeleteEventDetails`) and `UpdateStructureData` —
which are not overridden, so the SDK base rejects them. A backend exception is `BadInternalError` for that node and a deadline is `BadTimeout`.
No `AuditHistoryUpdateEvent` is emitted.

### The `HistoryWrite` AccessLevel bit

The SDK base admits a node to HistoryUpdate only if its `AccessLevel` carries `HistoryWrite`. The server ORs that bit in **only when the node is
historized *and* an updater is wired at materialise time** (the raw node and every UNS mirror; the surgical `UpdateTagAttributes` path recomputes
it the same way). `OtOpcUaServerHostedService.StartAsync` therefore wires the updater **before** installing the sink, and nodes materialised
earlier are never retro-updated. With `Backend=Gateway` the bit is never set.

### Authorization

HistoryUpdate is gated in **both** ACL modes (rewriting recorded history was never ungated). `WriteOperate` is deliberately **not** reused.

| `Security:Acl:Enabled` | Decision |
|---|---|
| `false` (default) | The session's API key is re-checked first (`AuthorizeKeyUsable`, Gitea #563 — an expired or disabled key loses its activation-time roles), then the session must carry the **`HistoryUpdate` role**. `ApiKeyInterimRoleMapper` confers it **only from a `Cluster`-scoped grant** carrying `NodePermissions.HistoryUpdate`; narrower-scoped grants confer nothing (the same under-grant rule as `WriteOperate`). Denied ⇒ `BadUserAccessDenied`. A `WriteOperate`-only key is denied. |
| `true` | `DataPlaneAuthorizer.Authorize(identity, OpcUaOperation.HistoryUpdate, nodeId, realm)` — the scoped `NodeAcl` grant decides per node and **replaces** the role check (never chained), so a session with no roles succeeds on the strength of a scoped grant. Denied ⇒ `BadUserAccessDenied` for that node. |

`NodePermissions.HistoryUpdate` (bit 12) is authorable in the AdminUI ACL editor and is **not** part of any bundle
(`ReadOnly` / `Operator` / `Engineer` / `Admin`) — it must be granted explicitly. See [security.md](security.md#data-plane-authorization).

---

## Authoring workflow

1. Open the equipment's **Tags** tab on `/uns/equipment/{id}`.
2. Create or edit the tag.
   - For **typed-editor drivers** (Modbus, S7, OpcUaClient, etc.): check the
     **Historize this tag** checkbox and, if needed, fill in the **Historian tagname
     (override)** textbox.
   - For **raw-JSON editors** (Galaxy): you can check the same first-class checkboxes
     (they appear below the JSON textarea), or add `"isHistorized":true` (and optionally
     `"historianTagname":"..."`) directly in the textarea.
3. Save and publish. The server rebuilds its address space; the node materialises with
   `Historizing=true` and the `HistoryRead` AccessLevel bit.
4. Confirm with Client.CLI `read` that the node's `Status` is `Good` and that the value
   is updating. Then issue a `historyread` to verify the historian connection returns data.

### Native-alarm historian opt-out (`alarm.historizeToAveva`)

A tag carrying a native `"alarm"` object has a **separate** historian opt-out for its
**alarm transitions** (distinct from tag-value historization). On the Tag modal, check or
uncheck **Historize to AVEVA** in the alarm section. This maps to `alarm.historizeToAveva`
(`bool?`) in the `TagConfig` JSON:

- **Absent or `true`** (default) — the alarm's transitions are written to AVEVA Historian
  via `HistorianAdapterActor`.
- **`false`** — the durable AVEVA write is suppressed for this alarm's transitions. The
  live `/alerts` feed and OPC UA condition events are unaffected.

The gate is applied in `HistorianAdapterActor` using `is not false` semantics, matching the
scripted-alarm `HistorizeToAveva` posture. See
[ScriptedAlarms.md §Native driver alarms](ScriptedAlarms.md#native-driver-alarms-equipment-tag-path)
and [AlarmTracking.md §Historian write-back](AlarmTracking.md#historian-write-back-non-galaxy-alarms)
for the full alarm-historian routing.

---

## Client.CLI historyread examples

The `historyread` command reads historical data from any node. Supply start and end times in
ISO 8601 UTC form. See [docs/Client.CLI.md](Client.CLI.md) for the full flag reference.

> **v3 Batch 4 NodeIds.** A historized tag is readable through **either** of its two NodeIds —
> the Raw-namespace node (`s=<RawPath>`, e.g. `Plant/Modbus/dev1/Speed`) or a referencing
> equipment's UNS-namespace node (`s=<Area>/<Line>/<Equipment>/<EffectiveName>`). Both register
> the same historian tagname, so the returned series is identical. The `ns=N` index is assigned
> at connect time per the namespace URI (`https://zb.com/otopcua/raw` /
> `https://zb.com/otopcua/uns`) — resolve it from the server's `NamespaceArray` rather than
> hard-coding it. The examples below show a Raw-namespace read (`ns=2` illustrative).

```bash
# Raw history for a historized tag via its Raw-namespace NodeId (last 24 hours by default)
otopcua-cli historyread \
  -u opc.tcp://localhost:4840/OtOpcUa \
  -n "ns=2;s=Plant/Modbus/dev1/Speed" \
  --start "2026-06-13T00:00:00Z" --end "2026-06-14T00:00:00Z"

# Same series via a referencing equipment's UNS-namespace NodeId, limited to 100 values
otopcua-cli historyread \
  -u opc.tcp://localhost:4840/OtOpcUa \
  -n "ns=3;s=filling/line1/station1/Speed" \
  --start "2026-06-13T00:00:00Z" --end "2026-06-14T00:00:00Z" \
  --max 100

# 1-hour average aggregate
otopcua-cli historyread \
  -u opc.tcp://localhost:4840/OtOpcUa \
  -n "ns=2;s=Plant/Modbus/dev1/Speed" \
  --start "2026-06-13T00:00:00Z" --end "2026-06-14T00:00:00Z" \
  --aggregate Average --interval 3600000

# Authenticated read (ReadOnly role or higher required)
otopcua-cli historyread \
  -u opc.tcp://localhost:4840/OtOpcUa \
  -n "ns=2;s=Plant/Modbus/dev1/Speed" \
  --start "2026-06-13T00:00:00Z" --end "2026-06-14T00:00:00Z" \
  -U reader -P password
```

Supported `--aggregate` values: `Average`, `Minimum`, `Maximum`, `Count`, `Start`, `End`,
`StandardDeviation` (aliases: `avg`, `min`, `max`, `stddev`/`stdev`, `first`, `last`).
`--interval` is the processing interval in milliseconds (default 3600000 = 1 hour).
The `Total` aggregate is served by the server's OPC UA HistoryRead endpoint for any conformant
client (derived as time-weighted Average × interval-seconds — see "Total aggregate derivation"
above), but is not exposed by this bundled CLI.

---

## Live /run gate

The live read gate requires a reachable `ZB.MOM.WW.HistorianGateway` (VPN to `wonder-sql-vd03`) with the
AVEVA Historian behind it healthy. Set `ServerHistorian:Enabled=true` with the correct `Endpoint`
(`https://host:5222`) and supply `ServerHistorian__ApiKey` via the environment, then deploy and publish at
least one historized Galaxy tag. The gate is operator-driven — it is not part of the local docker-dev rig.
The gateway-backed driver also ships an env-gated live suite (`Category=LiveIntegration`); see the
`HISTGW_GATEWAY_ENDPOINT` / `HISTGW_GATEWAY_APIKEY` / `HISTGW_TEST_TAG` / `HISTGW_WRITE_SANDBOX_TAG` /
`HISTGW_BOOL_SANDBOX_TAG` / `HISTGW_ALARM_SOURCE` env vars (it skips cleanly when they are absent).
`HISTGW_BOOL_SANDBOX_TAG` (e.g. `HistGW.LiveTest.BoolSandbox`, a **dedicated** writable Boolean tag —
never the shared Float write sandbox) drives the Boolean historization leg
(`EnsureTags_boolean_sandbox_provisions_as_Int2`) and the retype probe
(`EnsureTags_type_change_outcome_is_observable`) that back the
[Boolean historization maps to Int2](#boolean-historization-maps-to-int2-issue-441) decision. Running
the full suite against the gateway (all six tests, none skipped) also discharges 06/U-2's
"full documented run" for the historian suite.

See [AlarmHistorian.md](AlarmHistorian.md) for the alarm write path and
[ServiceHosting.md](ServiceHosting.md) for the (external) HistorianGateway deployment.

---

---

## Cross-driver TagConfig key reference

The `TagConfig` JSON blob is the single extension surface for per-tag server-side
behaviours across all drivers.  Keys from multiple phases coexist in the same blob;
unknown keys are preserved byte-stable on round-trip by all typed editors.

### Historian + alarm keys (Phase B / Phase C)

| Key | Type | Phase | Description |
|---|---|---|---|
| `isHistorized` | bool | C | Marks the tag historized (HistoryRead + Historizing=true). |
| `historianTagname` | string | C | Explicit historian tagname override (defaults to `FullName`). |
| `alarm` | object | B | Native alarm definition (`alarmType`, `severity`, `historizeToAveva`, …). |

### Array keys (Phase 4c)

| Key | Type | Phase | Description |
|---|---|---|---|
| `isArray` | bool | 4c | When `true` and `arrayLength >= 1`, materialises the node as a 1-D array (`ValueRank=OneDimension`). Absent or `false` → scalar. |
| `arrayLength` | uint (≥1) | 4c | Element count; sets `ArrayDimensions`. Required when `isArray: true`. |

### Cross-driver array coverage matrix

| Driver | Read mechanism | Live-verify status |
|---|---|---|
| **Modbus** | Contiguous FC03/FC04 block; String/BitInRegister modes supported | Mac-verifiable (sim `10.100.0.35:5020`) |
| **S7** | `ReadBytesAsync` block + per-element decode loop | Unit-proven (fixture down) |
| **AB CIP** | libplctag native array read (atomic + UDT member arrays) | Unit-proven (fixture down) |
| **AB Legacy** | PCCC multi-element file read via libplctag (cap 256 elements) | Unit-proven (fixture down) |
| **TwinCAT** | Raw-bytes array read + per-element decode (native notifications carry the whole array) | Live-verified on TwinCAT 2.11 (2026-10-07) |

Array writes, multi-dimensional arrays (ValueRank>1), and per-element historization are out of scope — see [Uns.md §Array tags](Uns.md#array-tags-1-d).

---

## Closed backlog §2 items (Phase 4 / Phase 4b)

The following items from the backlog §2 (archived at
`docs/plans/2026-06-18-stillpending-backlog-audit.md`) were closed by earlier
phases and are recorded here so future audits don't re-flag them.

| Item | Closed by | Commit |
|---|---|---|
| Modbus Int64 / UInt64 OPC UA node `DataType` correction | Phase 4 | `bd8fee61` |
| `HistoryAggregateType.Total` — `Total` aggregate | Phase 4 (derived client-side as time-weighted Average × interval-seconds) | `5e27b5f7` |
| Historian poison alarm-event indefinite retry — dead-letter cap (task #437) | Phase 4 | `fcb38014` |

---

## See also

- [docs/plans/2026-06-14-galaxy-phase-c-historian-design.md](plans/2026-06-14-galaxy-phase-c-historian-design.md) — full design and implementation notes
- [AlarmHistorian.md](AlarmHistorian.md) — alarm write path; drains to the same HistorianGateway (`SendEvent`)
- [AlarmTracking.md](AlarmTracking.md) — OPC UA Part 9 alarm surface (event history source)
- [Client.CLI.md](Client.CLI.md) — full `historyread` flag reference
- [ScriptedAlarms.md](ScriptedAlarms.md) §"Native driver alarms" — the Phase B `alarm` object in `TagConfig` (parallel carrier)
- [Uns.md §Array tags](Uns.md#array-tags-1-d) — `isArray`/`arrayLength` keys, cross-driver coverage, and deferrals