lmxopcua/docs/Historian.md

Historian — server-side OPC UA HistoryRead / HistoryUpdate (pluggable backends)

Phase C wires server-side OPC UA HistoryRead for authored equipment tags flagged historized. The feature is driver-agnostic: any equipment tag (Galaxy, Modbus, OpcUaClient, or any other driver) can be marked historized; the server dispatches all history reads to the registered IHistorianDataSource — the HistorianGateway read client (GatewayHistorianDataSource, talking gRPC to the external ZB.MOM.WW.HistorianGateway via the ZB.MOM.WW.HistorianGateway.Client package). No EF migration is required; the historian flag rides in the existing schemaless TagConfig JSON blob alongside the Phase B alarm object. (The bespoke Wonderware TCP sidecar backend this replaced was retired — see drivers/Historian.Wonderware.md.)

The backend is pluggable (2026-10). ServerHistorian:Backend selects Gateway (the default — everything described below as "the gateway"), SqlServer (a plain SQL Server database the server reads and writes itself, which also supports OPC UA HistoryUpdate) or QuestDb (a QuestDB time-series database over the Postgres wire protocol; also supports HistoryUpdate, via soft delete). See Backend selection, SQL Server backend, QuestDB backend and HistoryUpdate. Wherever this page says "the historian" without qualification it means the selected backend; sections that only apply to one backend say so.

Design references: docs/plans/2026-06-14-galaxy-phase-c-historian-design.md (Phase C) and docs/plans/2026-10-09-pluggable-historian-design.md (pluggable backends).

---

Historized TagConfig schema

A tag is historized by setting the Historize this tag checkbox and optional Historian tagname (override) textbox in the Tag modal on the /uns equipment page Tags tab. These controls work for all drivers — typed editors (Modbus, S7, OpcUaClient, etc.) and the raw-JSON textarea (Galaxy) alike. The controls merge isHistorized / historianTagname into the existing TagConfig JSON blob via the TagHistorizeConfig helper, preserving all other keys byte-stable. Drivers that still use the raw-JSON editor (Galaxy) can also add the fields directly in the textarea.

Fields

Field	Type	Required	Description
isHistorized	bool	yes	Marks the tag historized. Materialises the OPC UA node with Historizing=true and the HistoryRead AccessLevel bit.
historianTagname	string	no	Explicit historian tagname to query. When absent or empty, defaults to the tag's driver FullName (the TagConfig.FullName).

Examples

Default historian tagname (uses FullName = TestMachine_002.TestFloat):

{"FullName":"TestMachine_002.TestFloat","isHistorized":true}

Explicit historian tagname override:

{"FullName":"TestMachine_002.TestFloat","isHistorized":true,"historianTagname":"Plant.Line1.Flow"}

A tag that is both historized and a native alarm:

{"FullName":"TestMachine_002.HiAlarm","isHistorized":true,"alarm":{"alarmType":"OffNormalAlarm","severity":700}}

---

Backend selection (ServerHistorian:Backend)

ServerHistorian:Backend (case-insensitive, trimmed; default Gateway) picks the provider (IHistorianBackendProvider) that builds all five historian seams — the HistoryRead data source, the continuous- historization value writer, the alarm-history writer, the tag provisioner and (optionally) the HistoryUpdate updater. HistorianBackendRegistry in the Host owns the name table. An unknown name fails host start with an OptionsValidationException that lists the allowed values (Gateway, SqlServer, QuestDb), regardless of whether anything consumes the historian. The existing enable gates (ServerHistorian:Enabled, AlarmHistorian:Enabled, ContinuousHistorization:Enabled) are unchanged — they simply get their objects from the selected provider.

	Gateway (default)	SqlServer	QuestDb
Storage	AVEVA Historian via the HistorianGateway sidecar	{Schema} tables in a SQL Server database you point it at	{prefix}sample / event / tag / sample_modified WAL tables in a QuestDB instance
Value kinds recorded	Double, Int64, Boolean (numeric analog only)	all: Double, Int64, Boolean, String, DateTime, plus Empty (a Bad sample with no value)	same as SqlServer
Status code stored	Sent as DA quality but discarded by the gateway (C-002); AVEVA stores every sample Good	Stored bit-for-bit (StatusCode int column)	Stored bit-for-bit (unsigned, in a LONG)
Comms-loss / driver-Bad publishes	Dropped, counted in DroppedUnsupportedKind	Stored as an Empty outage-marker sample carrying the Bad code (see below)	Stored as an Empty outage-marker sample, as on SqlServer
Processed aggregates	TimeAverage, Minimum, Maximum, Total (Average and Count: BadAggregateNotSupported, #679)	The seventeen in the aggregate table (incl. Part 13 time-weighted, #671)	The same seventeen, identical results
HistoryUpdate	No (BadHistoryOperationUnsupported)	Yes (Insert / Replace / Update / DeleteRaw / DeleteAtTime)	Yes — soft delete (tombstones), see below
Modified-value history (IsReadModified, #668)	No (BadHistoryOperationUnsupported)	Yes — SampleModified, atomic with the change	Yes — {prefix}sample_modified, written before the change (orphan-on-crash)
Alarm history	gateway SendEvent	Event table (insert-if-absent)	{prefix}event table (dedup upsert on (ts, event_id))
Provisioning (EnsureTags)	numeric types only; others skipped	MERGE upsert of every request, all types	Upsert into {prefix}tag, all types
Connection	Endpoint + ApiKey + TLS keys (top-level ServerHistorian:*)	ServerHistorian:SqlServer:*	ServerHistorian:QuestDb:*

Recording stays Primary-only (#537) and at-least-once with the #538 per-tag quarantine on every backend; HistoryRead is served from any node (the database is shared). HistoryUpdate is Primary-only since #674 (BadInvalidState on any other node — see HistoryUpdate). The historian connection string is not a ConfigDb string, and a driver-only node may carry it without breaking the Phase 4 "no ConfigDb" posture.

---

ServerHistorian configuration

The ServerHistorian section in appsettings.json controls the historian read path. Enabled defaults to false; when disabled, the server registers NullHistorianDataSource and all HistoryRead calls on historized nodes return GoodNoData (empty, not an error).

{
  "ServerHistorian": {
    "Enabled": false,
    "Backend": "Gateway",
    "Endpoint": "",
    "ApiKey": "",
    "UseTls": true,
    "AllowUntrustedServerCertificate": false,
    "CaCertificatePath": null,
    "CallTimeout": "00:00:30",
    "MaxTieClusterOverfetch": 65536,
    "HistoryReadBatchParallelism": 4,
    "MaxConcurrentHistoryReadBatches": 16,
    "HistoryReadDeadline": "00:01:00",
    "MaxTimeWeightedRawRows": 1000000,
    "SqlServer": {
      "ConnectionString": "",
      "Schema": "hist",
      "CommandTimeout": "00:00:30",
      "AutoCreateSchema": true
    },
    "QuestDb": {
      "ConnectionString": "",
      "TablePrefix": "hist_",
      "RetentionDays": null,
      "CommandTimeout": "00:00:30",
      "AutoCreateSchema": true,
      "VisibilityTimeout": "00:00:05"
    }
  }
}

Key	Type	Default	Description
Enabled	bool	false	Enable the selected backend's read client. false → NullHistorianDataSource (empty reads) and no HistoryUpdate.
Backend	string	Gateway	Gateway, SqlServer or QuestDb (case-insensitive). Unknown ⇒ host start fails. See Backend selection.
Endpoint	string	""	Gateway backend only. Absolute gateway URI, e.g. https://host:5222. Scheme selects transport (https:// = TLS, http:// = h2c plaintext). Required when Enabled.
ApiKey	string	""	Gateway backend only. The gateway peppered-HMAC key (histgw_<id>_<secret>) sent as Authorization: Bearer. Required when Enabled. Supply via env ServerHistorian__ApiKey.
UseTls	bool	true	Gateway backend only. Connect over TLS; must match the Endpoint scheme.
AllowUntrustedServerCertificate	bool	false	Gateway backend only. Accept a self-signed / untrusted server certificate (dev / on-prem only).
CaCertificatePath	string|null	null	Gateway backend only. PEM CA file pinning the gateway's TLS chain. Null/empty uses the OS trust store.
CallTimeout	TimeSpan	00:00:30	Gateway backend only. Per-call deadline applied to each unary gateway read. A read for a tag the historian does not yet know about — e.g. a freshly-historized tag that has not been provisioned (see Tag auto-provisioning) — can run to this full deadline before the gateway errors, so an unprovisioned-tag HistoryRead can block for ~30 s. Lower this to fail faster, at the cost of truncating legitimately long reads.
MaxTieClusterOverfetch	int	65536	(Node-manager paging; all backends — a SqlServer PK rules out tie clusters, so it is moot there.) Maximum samples the server will fetch in one shot to page through a tie cluster (multiple samples sharing one SourceTimestamp). A cluster larger than this ceiling fails BadHistoryOperationUnsupported. Raise to handle abnormally large tie clusters; the default covers all normal-data cases.
HistoryReadBatchParallelism	int	4	Max historized nodes served concurrently within one HistoryRead batch (arch-review 03/S3). The SDK's HistoryRead* override surface is synchronous, so each arm bridges once at its boundary; this turns per-node serving from sequential (N × CallTimeout worst case) into ⌈N / P⌉ × CallTimeout. Also caps the gateway-load multiplication a single client can command. ≤ 1 serves sequentially.
MaxConcurrentHistoryReadBatches	int	16	Process-wide limit on HistoryRead (and HistoryUpdate) batches served concurrently (arch-review 03/S3). A flood of history clients degrades to BadTooManyOperations rather than exhausting the SDK request-thread pool. With HistoryReadBatchParallelism this caps total in-flight gateway reads at MaxConcurrentHistoryReadBatches × HistoryReadBatchParallelism (default 16 × 4 = 64). A batch that cannot get a slot within min(HistoryReadDeadline, 5 s) fails every handle BadTooManyOperations.
MaxTimeWeightedRawRows	int	1000000	SqlServer / QuestDb only (#671). The most raw in-window samples one time-weighted aggregate read (TimeAverage, TimeAverage2, Total, Total2, Interpolative) may stream into the server-side evaluator — that family is not pushed down to SQL. The window query is capped at this + 1 rows; one row past the cap fails that node with BadTooManyOperations (not counted as a backend fault) instead of holding a pooled connection until HistoryReadDeadline and returning BadTimeout. Bound rows outside the window do not count; per-sample aggregates are not capped. 1 000 000 ≈ 11.5 days of a 1 Hz tag. Non-positive = unbounded (a Validate() warning).
HistoryReadDeadline	TimeSpan	00:01:00	Per-request wall-clock deadline for a HistoryRead (and HistoryUpdate) batch (arch-review 03/S3). A node still in flight at the deadline surfaces BadTimeout, so one request can never hold an SDK request thread longer than this regardless of node count. Non-positive = unbounded (a Validate() warning).
HistoryUpdateMaxFutureSkew	TimeSpan	1.00:00:00	#674. How far past the server clock a HistoryUpdate (Insert / Replace / Update) item's SourceTimestamp may lie; an item beyond now + skew is refused BadOutOfRange by the node manager before any backend sees it (every backend). Guards against a far-future typo (a 2062 timestamp). The live recorder is not bounded. Non-positive disables the bound (no warning — a deliberate choice). See HistoryUpdate.

ServerHistorian:SqlServer (read only when Backend=SqlServer; bound by HistorianBackendRegistry.BindSqlServer):

Key	Type	Default	Validation	Description
SqlServer:ConnectionString	string	""	Required (non-blank)	ADO.NET connection string for the historian database. Never the ConfigDb. Supply via ${secret:OT-HISTORIAN-SQL} / environment (ServerHistorian__SqlServer__ConnectionString) — never commit a real one. The database must already exist; AutoCreateSchema creates the schema and tables, not the database.
SqlServer:Schema	string	hist	Must match ^[A-Za-z_][A-Za-z0-9_]{0,127}$ (it is spliced into DDL/DML)	SQL schema that holds SchemaVersion, Tag, Sample, Event and the SampleRow/TimeRow table types.
SqlServer:CommandTimeout	TimeSpan	00:00:30	Must be > 0	Per-command timeout, rounded up to whole seconds (minimum 1 s). The SQL backend does not use CallTimeout.
SqlServer:AutoCreateSchema	bool	true	—	true: on first use apply any missing embedded script under a sp_getapplock (OtOpcUaHistorianSchema), so concurrent nodes serialise. false: only verify SchemaVersion >= 1; if behind, every historian call fails with a clear error naming the missing version (surfaced in the read client's health LastError, never a host crash). A failed bootstrap is retried on the next call, not latched.
SqlServer:RetentionDays	int?	unset	>= 1 when set	Days of history kept, on source time. Unset keeps everything. When set, driver nodes purge older hist.Sample / hist.Event rows — see Retention.
SqlServer:PurgeInterval	TimeSpan	01:00:00	> 0	How often the purge runs (first pass at start-up).
SqlServer:PurgeBatchSize	int	4000	>= 1	Rows per DELETE TOP (n); the default stays under SQL Server's ~5000-lock escalation threshold so a purge never takes a table lock.

ServerHistorian:QuestDb (read only when Backend=QuestDb; bound by HistorianBackendRegistry.BindQuestDb): six keys — ConnectionString (required), TablePrefix, RetentionDays, CommandTimeout, AutoCreateSchema, VisibilityTimeout. The full table, defaults and validation rules are in QuestDB backend → Configuration (and Configuration.md).

Validation rules (ServerHistorianOptionsValidator, driver-role nodes only; admin-only nodes never build historian singletons):

• Backend must name a registered backend — checked always, with no I/O, even when nothing is enabled.
• Everything else is consumer-gated: it runs only when ServerHistorian:Enabled or AlarmHistorian:Enabled is true (the alarm sink sources its connection from this section even when ServerHistorian is off).
  • Backend=Gateway: Endpoint must be an absolute http(s) URI, ApiKey non-empty, and UseTls must agree with the endpoint scheme.
  • Backend=SqlServer: the three SqlServer rules above. Endpoint/ApiKey/UseTls are not required.
  • Backend=QuestDb: QuestDbHistorianOptions.Validate() (connection string required, prefix regex, RetentionDays >= 1, CommandTimeout > 0, VisibilityTimeout >= 0). Endpoint/ApiKey/UseTls are not required. A malformed TablePrefix is also rejected by the connection helper when the provider first builds a seam.
• ⚠️ Backend=SqlServer with a malformed Schema throws ArgumentException while the provider is built, so it stops host start even when no consumer is enabled (the validator is consumer-gated, provider construction is not).
• The alarm-only warning at startup (AlarmHistorian on, ServerHistorian off) names the selected backend.

Do not commit ApiKey to appsettings.json. Set it via the environment variable ServerHistorian__ApiKey, a secrets store, or a deployment-time overlay. The checked-in default is always empty.

Gateway-side prerequisites. The target gateway must run RuntimeDb:Enabled=true (continuous WriteLiveValues) + RuntimeDb:EventReadsEnabled=true (alarm-history ReadEvents), and the API key must carry the scopes historian:read, historian:write, historian:tags:write.

Migration from the Wonderware backend. Rename the old keys: Host/Port → Endpoint (https://host:5222); SharedSecret → ApiKey (env ServerHistorian__ApiKey); ServerCertThumbprint → CaCertificatePath (+ UseTls / AllowUntrustedServerCertificate).

The ServerHistorian section is independent of the AlarmHistorian section (the alarm write path) and the ContinuousHistorization section (driver-value capture). All three target the same gateway — but only ServerHistorian carries the connection (endpoint/key/TLS); the other two source it from there.

Continuous-historization value quality on the Gateway backend is still server-stamped. The recorder now carries each sample's real OPC UA status (it used to hard-code Good 192, so a Bad sample was historized as Good). GatewayHistorianValueWriter maps that status to an OPC-DA quality byte with GatewayQualityMapper.ToDaQuality (the inverse of the read-side table; unknown codes fall back by severity: Good ⇒ 192, Uncertain ⇒ 64, Bad ⇒ 0). But the gateway's WriteLiveValues SQL path discards the quality it is sent (a server-managed column; 0.2.0 Contracts C-002), so AVEVA still stores every recorded value as Good — a pre-existing limitation of that backend, now documented rather than fixed (archreview 06/C-7). The SqlServer backend stores the real status.

✅ Continuous value capture is LIVE-PROVEN END-TO-END (#491 CLOSED — gate run 2026-08-01, completed 2026-08-04; see docs/plans/2026-08-01-continuous-historization-live-gate.md). Against a real ZB.MOM.WW.HistorianGateway talking to the real AVEVA Historian on wonder-sql-vd03: deploy → EnsureTags provisioning → dependency-mux value delta → FasterLog outbox append → drain → WriteLiveValues → value read back from the historian, plus restart convergence (the recorder re-registers the deployed ref set after a process restart, with no new deployment).

The read-back used two registers carrying distinct non-zero constants (8224, 120), each of which read back as its own value — a 0-valued source cannot distinguish a correct value from a wrong one.

⚠️ The gate found a production-fatal defect: the recorder threw NotSupportedException on the first value it ever appended, because it touched Self after an await …ConfigureAwait(false), which discards Akka's ActorContext. Supervision restarted the actor, and since the recorder is seeded with an empty ref set, the restart discarded every historized ref — historization stopped after one value. Fixed; pinned by a regression test whose outbox fake completes on a pool thread (a fake returning ValueTask.CompletedTask, or even await Task.Yield(), cannot reproduce it).

The gate also found the two defects below, both since fixed.

Related, separately open: EnsureTags provisioning fires only for newly-added historized tags (AddressSpaceApplier iterates the added set), so flipping an existing tag to "isHistorized": true never provisions it — see docs/plans/2026-06-27-historian-gateway-integration-issues.md Issue 5.

---

ContinuousHistorization configuration

ContinuousHistorizationOptions binds the ContinuousHistorization section. It carries only the recorder + outbox knobs — the backend connection (gateway endpoint / key / TLS, or the SQL Server connection string) is sourced from ServerHistorian (above), never restated here. The outbox is a node-local FasterLog and is registered the same way for every backend.

Key	Type	Default	Description
Enabled	bool	false	true and a configured ServerHistorian wires the FasterLog outbox + the selected backend's value writer and spawns the ContinuousHistorizationRecorder. false → no recorder; driver tag values are not historized.
OutboxPath	string	""	Directory holding the FasterLog segment + commit files. Required when Enabled. In production set an absolute path on durable storage — a relative path resolves against the host's working directory.
CommitMode	string	PerEntry	PerEntry fsyncs before the append returns (no loss window); Periodic batches commits onto a background timer every CommitIntervalMs. Parsed case-insensitively; an unrecognized value falls back to PerEntry. See the batch-commit note below — PerEntry is one fsync per ingest burst, not per value.
CommitIntervalMs	int	100	Periodic-mode commit cadence (ms); must be > 0 under Periodic, ignored under PerEntry.
DrainBatchSize	int	1024	Outbox entries peeked + written per drain pass. Raised from 64: the peek is an in-memory scan of the outbox's live index (widening it costs allocation, not I/O), and a pass is now bounded-parallel over DrainConcurrency rather than sequential, so a wide batch no longer costs tags × latency. At 64 a drain drew down at most 64 entries every DrainIntervalSeconds — below the arrival rate of even a modest historized tag set — so any gateway hiccup left the outbox permanently behind, growing toward Capacity and shedding oldest samples instead of catching up.
DrainIntervalSeconds	double	2	Steady drain cadence (and the post-success reschedule).
DrainConcurrency	int	8	Historian tags a single drain pass writes concurrently. One peeked batch is grouped per historian tag and each group is an independent gateway round-trip, so a sequential pass cost tags × latency; this window makes it ⌈tags / concurrency⌉ × latency while bounding how many round-trips can be outstanding against the gateway at once. 1 restores the fully sequential drain; a value < 1 is clamped to 1 (with a Validate() warning). The ack is unchanged — still all-or-nothing after every group has reported, and the #538 per-tag quarantine still keys on "another group wrote this pass".
Capacity	int	500000	Max un-acked outbox entries before the drop-oldest policy sheds the front of the queue; 0 = unbounded. Roughly 50 MB of FasterLog. The #538 disk backstop for the one case the per-tag quarantine deliberately cannot resolve — a historian unreachable or refusing every write, where nothing proves any particular tag is at fault so the recorder correctly retains.
MaxWriteAttempts	int	5	#538 poison-tag budget: consecutive failed drain passes one historian tag may accumulate before its queued entries are abandoned and the FIFO head is allowed to advance. Counted only across passes where other tags wrote successfully, so an ordinary gateway outage never advances it. Clamped to at least 1; each abandonment logs a Warning naming the tag.
CaptureWhilePrimaryUnreachable	bool	true	#672: a standby whose capture is stood down for its mesh-partner Primary (#537) resumes capturing as soon as the cluster failure detector reports that Primary unreachable, instead of waiting for auto-down + a fresh redundancy snapshot. Shrinks a crash-failover capture gap from ~25 s to the failure-detector window (~10–12 s), at the cost of short duplicate windows — see the crash-failover note. false restores the pre-#672 behaviour. Never affects the alarm drain or the Primary data-plane gate.
MinBackoffSeconds	double	1	Initial retry backoff after a failed drain pass.
MaxBackoffSeconds	double	30	Cap on the exponential retry backoff.

PerEntry now means one fsync per ingest burst, not per value. The recorder appends a whole value burst through IHistorizationOutbox.AppendBatchAsync, which enqueues every entry and then issues a single group commit — so a burst of N values costs one fsync rather than N. The durability contract is unchanged: the call does not return until that commit completes, so nothing the caller has been told is buffered can be lost to an unclean restart. (The documented residual is unchanged too: a throw before the commit leaves the enqueued prefix tracked-but-uncommitted — served correctly for the life of the process, absent after an unclean restart — exactly the enqueue-then-commit window the single-entry path always had, widened from one entry to the batch.)

Cross-repo follow-up — a multi-tag batch WriteLiveValues on the gateway contract. The drain's unit of work is one gateway round-trip per historian tag, because ZB.MOM.WW.HistorianGateway.Client 0.3.0 exposes no RPC that writes values for more than one tag in a call. Bounded parallelism (DrainConcurrency) is therefore the ceiling: it overlaps the round-trips but cannot remove them, so a wide tag set still pays ⌈tags / concurrency⌉ network hops per drain pass. Collapsing that to a single hop needs a batch/multi-tag WriteLiveValues on the historian_gateway.v1 contract in the ~/Desktop/HistorianGateway sister repo. Until that lands, raising DrainBatchSize widens each hop's payload but not the hop count — which is why the concurrency window, not the batch width, is what governs drain throughput today.

Typed values, real status and outage markers

The recorder no longer assumes "a numeric Good sample". Each mux delta is coerced to a typed HistorianValue (Double / Int64 / Boolean / String / DateTime / Empty; a ulong above long.MaxValue becomes Double; arrays and anything else are dropped and counted) and appended with the real status the driver published. A value whose kind is not in the selected backend's SupportedKinds is dropped at capture and counted in RecorderStatus.DroppedUnsupportedKind (the field was DroppedNonNumeric before the rename; Empty is supported by SqlServer, not by Gateway).

⚠️ Comms loss is historized — as an outage marker (design §9.10). DriverHostActor.FanCommsLossToDependencyMux (#481) fans (null, BadNoCommunication) for every value ref of a comms-lost driver to the dependency mux, and the recorder cannot tell that from a driver's own Bad publish. So:

• SqlServer and QuestDb: each disconnect stores one Empty / BadNoCommunication (0x80310000) sample per historized tag. A driver's own Bad publish is stored the same way (Empty kind plus its own code — e.g. Modbus BadCommunicationError 0x80050000). This is deliberate: raw reads show the gap, and aggregates and AtTime interpolation do not bridge it. Cost: one extra row per tag per outage.
• Gateway: the marker is Empty, which that backend does not support, so it is dropped and counted in DroppedUnsupportedKind.

Outbox format v2 and rollback

Outbox entries are written in format v2: "HV2\0" magic + version byte 2 + id + tag + value kind + a kind-specific payload + the 32-bit status + timestamp. Entries left in an outbox by an older build (the fixed 38-bytes-plus-tag layout, always quality 192) are still read — as a Double with status Good. A record that matches neither layout is skipped and counted, never thrown, so a corrupt record cannot jam the drain (this also holds during the FasterLog recovery scan at start-up); the lifetime count is the gauge otopcua.historian.outbox.unreadable (IHistorizationOutbox.UnreadableCount). A growing value means undecodable records are being skipped.

Metric prefixes differ. The outbox's other two gauges are otopcua.historization.outbox.depth and otopcua.historization.outbox.dropped; the unreadable gauge is otopcua.historian.outbox.unreadable (the name the design fixed). All three describe the same outbox — search for both prefixes.

⚠️ Rollback: drain or delete the outbox before downgrading. A node running a pre-v2 build cannot read v2 records — its outbox deserializes every record in its constructor and fails to start on the first v2 one. Before rolling a node back, let the outbox drain to zero (otopcua.historization.outbox.depth) or stop the node and delete the directory named by ContinuousHistorization:OutboxPath (accepting the loss of anything un-drained). Upgrading needs no action.

Delivery semantics on SqlServer (QuestDb is the same, at µs resolution and with tombstones counting as present — see QuestDB backend). Writes are insert-if-absent on (TagId, SourceTimeUtc): the outbox's at-least-once redelivery (and both pair nodes writing the same sample in a boot window) converges on one row, and the first-stored sample wins. Duplicate timestamps inside one batch collapse to the first. A NaN / ±Infinity Double cannot be stored in a SQL float, so the value writer stores it as an Empty sample — the tag is not quarantined for one bad reading (HistoryUpdate behaves differently: see below). ⚠️ An Empty sample is always Bad (the quality contract: a stored sample is never served better than "no value"): a non-finite Double that arrived with a Good or Uncertain status is stored as Empty + BadOutOfRange (0x803C0000), and any other Empty with a non-Bad status as Empty + BadNoData (0x809B0000). An existing Bad code — the comms-loss BadNoCommunication marker, a driver's own Bad publish — is stored as it is. A duplicate-key error (2627/2601) rolls the whole statement back, so the writer re-runs the statement (at most twice) instead of treating the error as success.

---

A poison tag no longer pins the outbox (#538)

DrainOnceAsync peeks a batch, writes every per-tag group in it, and then acks all-or-nothing — because the outbox's RemoveAsync truncates the FIFO head through the acked id, so acking one tag's entries could silently drop older, unwritten entries belonging to another. That much is deliberate and unchanged.

What was wrong is what happened when a tag failed permanently: the head never advanced, so every healthy tag in the batch was re-submitted to the historian on every drain pass, indefinitely — duplicate samples at identical timestamps, and an outbox that never truncated (observed live at 8.3 MB from a single unprovisionable tag, with the gateway's ok and error counters climbing together — the signature of this bug).

The drain now tracks consecutive failures per historian tag and abandons a tag's queued entries once it exceeds ContinuousHistorization:MaxWriteAttempts (default 5), letting the head advance. Two properties matter:

• A pass where NOTHING wrote abandons nothing. The counter only advances on a pass where at least one other tag wrote successfully — that is what distinguishes "this tag is poison" from "the backend is down". A count-only quarantine looks identical on the poison-tag case and silently discards the whole backlog during an ordinary gateway restart, turning a duplicate-samples bug into a lost-history one. Pinned by a falsification-control test (A_backend_that_fails_every_tag_abandons_nothing) that goes red when the evidence check is removed while the poison-tag test stays green.
• A tag over budget waits for the others. Abandonment is deferred until every failing tag in the batch is over budget, so a tag is never discarded on a pass that could not have advanced the head anyway.

Each abandonment logs a Warning naming the tag and the discarded sample count, and the total rides RecorderStatus.AbandonedEntries. The most common cause is a tag the historian has no definition for — easy to reach, because EnsureTags provisioning runs only for newly-added historized tags, so a changed historianTagname is never provisioned (Issue 5 above).

ContinuousHistorization:Capacity now also defaults to 500,000 entries rather than unbounded. It is the disk backstop for the one case the quarantine deliberately cannot resolve — a historian refusing every write, where retaining is correct. Set 0 to restore unbounded retention.

Only the recording Primary captures (#537)

Both nodes of a redundant pair serve the same address space and see the same dependency-mux fan-out, so an ungated recorder wrote every sample twice (measured: the gateway's write counter advanced twice per value with both centrals up, and halved when one was stopped).

The recorder now reads a capture gate per value, fed from PrimaryGatePolicy.ShouldRecordHistorizedValues via IHistorizationRoleView — the same shape as the alarm store-and-forward sink's drainGate, with two deliberate differences:

• It gates the CAPTURE, not the drain. The alarm queue lives in LocalDb and replicates, so the Secondary already holds the Primary's rows and gating the drain loses nothing. The historization outbox is a node-local FasterLog replicating to nobody: gating only its drain would leave the standby appending forever to a queue no one empties — unbounded disk, and a flood of stale duplicates the moment it is promoted. Entries a node already captured are still drained by that node, so a demotion loses nothing recorded.
• The relevant peer is one that records the same refs, not one that holds the same rows. A node stands down only for a Primary in its own per-cluster mesh. A node carrying no cluster-scoped role cannot establish that, so it never stands down: under a fleet-wide election the elected Primary is usually running a different cluster's configuration, and deferring to it would end this cluster's historization silently — the drain would report success on an empty outbox forever. That is the same scope mismatch that once suspended the alarm drain on every node of the docker-dev rig at once.

An unknown role captures, matching the alarm drain: a duplicate sample is recoverable, a hole in the trend is not. The residual cost is a capture gap across a failover, bounded by how long the newly promoted node takes to see its first redundancy snapshot — inherent to any non-duplicating design. Suppressed values are counted in RecorderStatus.SuppressedNotPrimary, and each gate transition logs once.

A crashed Primary: capture gap (#672)

On a crash (SIGKILL / docker kill) that "first snapshot" can be a long way off. The dead Primary stays a cluster member — unreachable, not downed — for the failure-detector window plus auto-down-unreachable-after (acceptable-heartbeat-pause 10 s + 15 s in akka.conf), and the redundancy singleton almost always lived on it (it is the oldest member), so no snapshot is published until auto-down re-homes the singleton. The standby's cached snapshot keeps naming the dead node Primary and nobody captures for ~30 s (measured on the QuestDB live gate).

DriverHostActor therefore follows cluster reachability (its own ClusterEvent.ReachabilityEvent subscription) and, with ContinuousHistorization:CaptureWhilePrimaryUnreachable on (the default), excludes an unreachable peer from the historization "a mesh-partner Primary is recording" question, re-deriving and republishing only the capture verdict from the last snapshot. A node at the same address coming Up again clears the mark; a reachable-again Primary closes the gate again. It logs capturing historized values while recording Primary <id> is unreachable once on entry and once on revert. The alarm drain and the Primary data-plane verdict (writes, native acks, the Sparkplug host claim) are not touched — a false allow there is a dual claimant, not a duplicate sample.

Trade-offs, all in the duplicate direction #537 already accepts:

• The gap shrinks to the failure-detector window (~10–12 s), not to zero — detection is the floor.
• Duplicates during the overlap. A Primary that is only briefly unreachable (GC pause, network blip) overlaps the standby until it is reachable again, and under a genuine partition both sides capture from detection onward (as they already did after auto-down, now ~15 s earlier). SqlServer and QuestDB absorb a duplicate of a source-timestamped sample idempotently ((TagId, SourceTimeUtc) key / dedup), but a driver with no source timestamp (e.g. Modbus: SourceTimestampUtc is null, so each node stamps its own poll time) writes a duplicate row on every backend, and the Gateway backend stores duplicates of everything.
• Set CaptureWhilePrimaryUnreachable=false to keep the pre-#672 behaviour (no duplicates from this path, the full ~30 s hole on a crash).

---

Tag auto-provisioning (EnsureTags)

When a deploy adds a historized tag, the server auto-ensures it exists in the historian via the gateway's EnsureTags (Gateway) or the Tag-row MERGE (SqlServer) before the first value or HistoryRead lands. AddressSpaceApplier.Apply() fires a non-blocking, fire-and-forget IHistorianProvisioning.EnsureTagsAsync for each added historized value tag (with the resolved historian name — the historianTagname override when set, else the driver FullName); the hook is wrapped so a faulted, slow, or throwing provisioner can never block or fail a deploy. On the Gateway backend, non-numeric (String/DateTime/Reference) types are skipped, not provisioned; the SqlServer backend provisions every type (below).

Wiring + gating. The concrete provisioner comes from the selected backend's provider (CreateProvisioning): the gateway-backed GatewayTagProvisioner, or — for SqlServer — SqlServerTagProvisioner (one MERGE … WITH (HOLDLOCK) upsert per chunk of 400 tags; every data type is provisionable so skipped is always 0; a blank or over-400-character name fails alone; a null unit/description in a request keeps the stored value). It is registered by the Host's AddHistorianProvisioning and gated on the same ServerHistorian:Enabled flag as the read path — both use the one resolved backend. When the section is disabled (or absent) the applier resolves the no-op NullHistorianProvisioning and provisioning does not run (so on a read-only-historian deployment, ensure tags exist by some other means). On the Gateway backend the API key must carry the historian:tags:write scope for EnsureTags to succeed.

Observability. Every dispatch logs an Information tally — historian provisioning completed (dispatched=N, requested=…, ensured=…, skipped=…, failed=…). A line showing dispatched=N but requested=0 indicates the no-op provisioner ran (provisioning is not wired for this node), which is the signal to check that ServerHistorian:Enabled=true.

Why this matters for HistoryRead latency. Until a tag is provisioned, the historian does not know it, so a HistoryRead on it runs to the full CallTimeout (~30 s) before erroring. Auto-provisioning on deploy keeps freshly-historized tags from hitting that slow path. See the CallTimeout row above.

Deferred types and the UInt16 fallback (#600 item 4)

Gateway backend only. The SqlServer backend provisions and records every type (String and DateTime included).

Historization on the Gateway backend is numeric-analog only. HistorianTypeMapper classifies exactly nine types as historizable (Boolean, Int16/32/64, UInt16/32/64, Float32, Float64):

• Deferred: String, DateTime, Reference. The historian's SQL analog write path cannot carry them. Provisioning skips them (counted in the skipped= tally, via IsHistorizable), the mapper throws NotSupportedException for them, and ContinuousHistorizationRecorder drops their values and counts them in DroppedUnsupportedKind (renamed from DroppedNonNumeric). A tag authored isHistorized with one of these types therefore silently records nothing; HistoryRead on it returns whatever the historian holds (typically GoodNoData).
• Fallback: UInt16 -> Uint4. The historian's UInt2 write path is unproven upstream, so UInt16 tags are provisioned as the wider UInt4. Values are unaffected; only the historian-side storage type differs. (Boolean -> Int2 is the other fallback, below.)

Lifting either needs gateway-side support first; neither is scheduled.

Boolean historization maps to Int2 (issue #441)

EnsureTags is contract-documented (HistorianGateway 0.2.0+) as an upsert — "creates or updates historian tags according to the given definitions" — that maps per-tag onto AVEVA's HistorianClient.EnsureTagAsync(definition) -> bool. It is not idempotent (not routed through the idempotent retry pipeline) and not skip-if-exists.

HistorianTypeMapper maps DriverDataType.Boolean → HistorianDataType.Int2 and Boolean values historize as a small 0/1 integer. Int2 is used rather than the semantically-exact Int1 because the historian's Int1 analog-tag creation path is server-degenerate — EnsureTags(Int1) stores an unusable stub, so a Boolean tag mapped to Int1 never becomes historizable. This is an evidence-based decision from a live probe against the deployed 2023 R2 historian; the root cause and gateway-side resolution are tracked in issue #441 and HistorianGateway #11 / PR #16 (which documents Int1 as an explicit, evidence-based exclusion). The value-write path is unaffected: the recorder sends every value as a double on WriteLiveValues regardless of tag type, so a Boolean 0.0/1.0 round-trips cleanly into the Int2 tag.

Observed live (2026-07-13, real 0.2.0 gateway → real AVEVA historian on wonder-sql-vd03, probed via grpcurl — archreview R2-06 T12):

• EnsureTags is a real create-or-update upsert — re-EnsureTags of an existing tag at the same type returns Success=true.
• An in-place retype between SUPPORTED types is ACCEPTED — Float→Double on an existing tag returns Success=true.
• A type sweep confirmed Int2/UInt2/Int4/UInt4/Float/Double all provision Success=true; only Int1 fails (Success=false, Error="ProtocolEvidenceMissingException" — the vendored client's "no capture evidence for this type" guard → gRPC Unimplemented), both for a fresh tag and as a retype target. Mapping Boolean to Int2 sidesteps that gap entirely.

Prerequisite for provisioning to land at all: the provisioner must stamp a positive StorageRateMs — a 0 made EnsureTags throw ArgumentOutOfRangeException and fail every provision (fixed in GatewayTagProvisioner, PR #439).

Migration note for pre-existing Boolean tags. A Boolean tag historian-side as Float (e.g. from the 0.1.0-era default) is asked, on its next deploy, to upsert to Int2. Both are supported analog types, so the retype is accepted. Provisioning is fire-and-forget and never blocks/fails a deploy; a per-tag rejection (if any) surfaces only as the tally's failed count, and values written to a still-Float tag remain readable (Float supersets the 0/1 range) pending reconcile. A bulk retype sweep is not done in code.

---

HistoryRead behavior

Read variants

The server supports all four OPC UA HistoryRead variants:

Variant	Node type	CLI --aggregate
Raw	Historized variable	(omit --aggregate)
Processed	Historized variable	Average, Minimum, Maximum, Count, Start, End, StandardDeviation (the CLI's own list; SqlServer / QuestDb serve seventeen aggregates — see the aggregate table)
AtTime	Historized variable	n/a (client supplies exact timestamps)
Events	Equipment folder (event notifier)	n/a
Modified (Raw with IsReadModified = true, #668)	Historized variable	--modified (see Modified-value history)

Variable nodes (historized tags) serve Raw, Processed, and AtTime history. Historizing=true and AccessLevels.HistoryRead are set at materialization so any compliant OPC UA client can discover historized capability from the node's attributes.

v3 Batch 4 — dual-namespace registration. A historized raw tag surfaces as two variable nodes: the Raw-namespace node (ns=Raw, s=<RawPath>) and, for each referencing equipment, a UNS-namespace node (ns=UNS, s=<Area>/<Line>/<Equipment>/<EffectiveName>). Both NodeIds register the SAME historian tagname (the raw tag's FullName / historianTagname), so HistoryRead against either NodeId resolves to the same tagname and returns the same series. The historization intent is single-sourced — the mux's HistorizedTagRef set stays keyed by RawPath (no doubles), and continuous historization / EnsureTags provisioning run once per raw tag regardless of how many equipment reference it.

Equipment-folder event-notifier nodes serve Event history. Every equipment folder that owns at least one alarm condition is already an event notifier; the server registers a sourceName (the equipment id) for each such folder and maps event history reads to the HistorianGateway using that source. (Alarm-history ReadEvents requires the gateway running RuntimeDb:EventReadsEnabled=true.) Event-field projection supports the standard BaseEventType select clauses — EventId, SourceName, Time, ReceiveTime, Message, and Severity; an unsupported select operand returns a null field (spec-conformant).

No driver-side history (Gitea #655)

History is served only by the server-wide IHistorianDataSource (GatewayHistorianDataSource / NullHistorianDataSource). No driver contributes history: the OpcUaClient driver does not proxy its upstream server's HistoryRead (the former IHistoryProvider passthrough was never dispatched to and was deleted). On an OpcUaClient tag, isHistorized means "OtOpcUa records this value into the historian from now on", not "forward history reads upstream". A node without isHistorized has no HistoryRead access bit and answers BadHistoryOperationUnsupported.

Graceful degradation

Situation	HistoryRead status
Historized node, historian configured and reachable, results found	Good
Historized node, historian configured, time range empty	GoodNoData
Historized node, historian NOT configured (Enabled=false / Null source)	GoodNoData (empty)
Non-historized node	BadHistoryOperationUnsupported
Backend timeout or exception	BadHistoryOperationUnsupported per node; other nodes in the same batch are unaffected
Backend=SqlServer, AutoCreateSchema=false, schema missing/behind	every read/write fails with a clear error naming the missing schema version (health LastError); reads answer BadHistoryOperationUnsupported per node; no host crash

A historized node with no historian configured never returns an error status — it returns empty. This means a deployment can author and publish historized tags before the historian gateway is provisioned, without producing error spikes in connected clients.

Continuation-point paging (Raw)

HistoryRead-Raw is paged server-side. The historian backend is single-shot (it returns up to NumValuesPerNode samples with no continuation point of its own), so the server synthesises paging time-based:

• A page that returns exactly NumValuesPerNode samples (with NumValuesPerNode > 0) MAY have more behind it, so the server stores a resume cursor and returns an opaque continuation point (16 bytes). The client hands it back to fetch the next page.
• A short page (fewer than the cap) is the last page — no continuation point.
• NumValuesPerNode == 0 ("all values, no limit") is never paged; the whole window returns in one shot.
• The resume cursor is tie-safe: the next page resumes from the last returned sample's SourceTimestamp inclusive and drops the boundary samples already emitted, so samples sharing the boundary timestamp are neither duplicated nor skipped.

Oversized tie clusters — within-timestamp paging. When more samples share one SourceTimestamp than the current page cap, the server detects that the cursor has stalled on a tie cluster (the last returned timestamp equals the resume timestamp). It then over-fetches the entire cluster at that single timestamp up to a bounded ceiling controlled by ServerHistorian:MaxTieClusterOverfetch (default 65 536), then serves the cluster NumValuesPerNode samples at a time across successive pages, advancing the cursor one tick past the timestamp once the cluster is fully drained.

Reverse reads (start > end, newest first) page the same way in the opposite direction: a drained cluster steps the cursor one tick down, the read ends when that would reach the (excluded) end, and the over-fetched cluster is sliced in descending order. Before the fix a reverse read with a small NumValuesPerNode stopped after its first page with GoodNoData.

Short pages within a cluster still carry a continuation point. A within-cluster page that returns fewer than NumValuesPerNode samples (because the cluster happened to be smaller than the cap, or is the final partial batch) is not the last page if the cluster itself has not been fully emitted — the server retains the continuation point so the client can drain the remainder. Only when the cluster is exhausted and the cursor has advanced past the timestamp does the short-page rule apply.

Cluster larger than MaxTieClusterOverfetch. If the over-fetch itself reaches the ceiling without spanning the full cluster, the node fails loudly with BadHistoryOperationUnsupported and the tag + timestamp + ceiling are logged. Remedies: raise MaxTieClusterOverfetch (or NumValuesPerNode) to cover the full cluster, or investigate the data anomaly (raw samples normally carry strictly increasing distinct timestamps).

For a single tag's raw history a tie cluster larger than the default 65 536 is a severe data anomaly. The ceiling exists to bound server-side memory on pathological data, not to cap normal operation.

Continuation points are bound to the OPC UA session (the SDK's ServerConfiguration.MaxHistoryContinuationPoints cap, default 100, with oldest-eviction; points are disposed when the session closes). Resuming an unknown / evicted / released point returns BadContinuationPointInvalid. releaseContinuationPoints drops the stored cursors without reading data.

Continuation-point paging (Events) (#669)

A capped event read (NumValuesPerNode > 0) pages exactly like Raw, with one difference: the trigger is the source's truncation flag (HistoricalEventsResult.ContinuationPoint != null), never the page count, so a cap-sized page the source says is complete carries no continuation point. NumValuesPerNode == 0 stays unbounded and never pages.

• The cursor is (source, next start, end, boundary skip, cap) in the same session-bound store as Raw (flagged as an events cursor — a Raw point presented to an events read, or the reverse, is BadContinuationPointInvalid). The next page reads from the last event's EventTimeUtc inclusive and drops the events at that timestamp already emitted — and asks the source for that many more rows than the cap, so every page short of the last is a FULL NumValuesPerNode page (#675; resumed pages used to come back short by the skip).
• A tie cluster larger than the cap is paged within by over-fetching the half-open window [T, T + resolution) up to MaxTieClusterOverfetch; beyond that bound the node answers BadHistoryOperationUnsupported (raise the bound).
• Tie exactness depends on the backend. SqlServer and QuestDB order by (EventTimeUtc, EventId), so the skip count is exact. The Gateway backend has no stable intra-timestamp order (and only a truncation flag), so ties at a page boundary are best-effort there.
• Reverse event reads page newest-first (#675). StartTime > EndTime reads the Part 11 reverse window (EndTime, StartTime] with events in DESCENDING (EventTimeUtc, EventId) order, so a cap keeps the latest N and a continuation point is issued when the source flags more. The direction is stored in the cursor: a reverse point resumes reverse (and a forward one forward) whatever the resume request's details say. Reverse tie clusters are over-fetched forward and sliced from the newest end, exactly as Raw does. Before #675 a capped reverse read silently returned the EARLIEST N, ascending, Good, with no continuation point.
• End-only events reads: backward on SqlServer/QuestDB, forward on Gateway (#675). StartTime unset (DateTime.MinValue) with an EndTime and NumValuesPerNode > 0 pages backward from EndTime over (MinValue, EndTime] — the same rule as Raw — only on a backend that reports SupportsReverseEvents (SqlServer, QuestDB). On Gateway (the default backend) and Null it keeps its pre-#675 forward [MinValue, EndTime) meaning, paged oldest-first, rather than becoming a refused reverse read. Start-only reads page forward to an open upper bound (DateTime.MaxValue, UTC) everywhere. Without a cap an end-only read is forward [MinValue, EndTime) everywhere.
• Backends. The server checks IHistorianDataSource.SupportsReverseEvents and then calls ReadEventsAsync(…, bool reverse, …). SqlServer and QuestDB serve reverse natively (… > lo AND … <= hi ORDER BY time DESC, id DESC) and report true. Gateway and Null report false: an explicit reverse read (StartTime > EndTime) is answered BadHistoryOperationUnsupported without calling the backend — the gateway streams ascending only and rejects end < start, and draining the whole window to keep its tail would be exactly the unbounded read the cap prevents. Because the refusal is a capability check, not a caught exception, a NotSupportedException thrown from inside a backend (a driver library) is logged as a backend fault like any other. Every implementation (and any future wrapper) must declare both members itself — the interface defaults (false / throw) would silently disable reverse reads; HistorianDataSourceReverseEventsGuardTests pins it.
• Gateway open upper bound. The DateTime.MaxValue (UTC) bound of a start-only forward read is pinned offline through the published client's proto Timestamp conversion (captured by an in-process gRPC server) and the FILETIME encoding (GatewayEventBoundsTests). What a live AVEVA Historian does with a year-9999 bound is still unverified (needs the VPN).
• No session on the request (in-process callers) degrades to single-shot, as for Raw.

Modified-value history (Part 11)

Gitea #668. On the SqlServer and QuestDb backends every HistoryUpdate leaves a modification trail: one row per sample it inserted, replaced or deleted, carrying the value as it was before the change (none for an insert), the caller's principal, the modification time and the Part 11 HistoryUpdateType. A HistoryReadRaw with IsReadModified = true returns that trail as a HistoryModifiedData — one DataValue (the prior value) plus one parallel ModificationInfo (ModificationTime, UpdateType, UserName) per entry.

Operation	Recorded UpdateType	Prior value
Insert	Insert	none — served null / BadNoData
Replace	Replace	the replaced value and its status
Update (upsert)	Update	the replaced value, or none when it inserted
DeleteAtTime / DeleteRaw	Delete	the deleted value and its status

• Only applied items are recorded. A refused item (BadEntryExists, BadNoEntryExists, BadInvalidTimestamp, …), a DeleteRaw over an empty range, the live recorder's writes and the retention purge leave nothing.
• UserName is the session's principal — the API key's NodeAcl principal, or anonymous for an admitted anonymous session (the key id is stored too, not served). Longer than 200 characters is truncated, never refused.
• Window and order. Same windows as Raw: forward [start, end) ascending by (SourceTime, ModifiedId), reverse (start > end) (end, start] descending, start == end the modifications at exactly that time; an unset bound with a cap pages open-ended like Raw. Several modifications of one timestamp appear in the order they were made.
• Paging. NumValuesPerNode > 0 pages with session-bound continuation points keyed on the last entry's (SourceTime, ModifiedId) — an exact, exclusive key, so there are no ties to over-fetch and no trailing empty page (the backend reads cap + 1 rows to know). The cursor is bound to the node and to the modified read kind (a raw or events point is BadContinuationPointInvalid). NumValuesPerNode == 0 is unbounded.
• Authorization is the ordinary HistoryRead decision — no new operation or permission (ungated with Security:Acl:Enabled off).
• Unsupported ⇒ BadHistoryOperationUnsupported: the Gateway backend (the AVEVA Historian exposes no modification trail) and the Null default. Deleting the trail (DeleteRawModifiedDetails.IsDeleteModified = true) is not supported either — the trail is an audit record and is aged out only by retention (below).
• Atomicity differs by backend. SqlServer writes the trail row with OUTPUT … INTO in the same statement that changes hist.Sample, so a change and its trail row commit or roll back together. QuestDB has no multi-table transaction, so the trail row is written before the change: a crash (or failed sample INSERT) in between leaves an orphan trail row describing a change that never landed — never a change without its trail row. Treat a QuestDB trail entry whose sample still shows the prior value as such an orphan. A failed sample write after its trail rows landed logs a Warning naming the tag, the row count and the time range, so the orphan can be found.
• Rolling upgrade (SqlServer). The v2 schema is applied by the first node that upgrades. Until its peer is upgraded too, that not-yet-upgraded node still runs the old build, whose HistoryUpdates write no trail rows — silent gaps in the audit trail. Upgrade both nodes of a pair together (stop, upgrade, start both); modifications made by an old-build node in the window are never recorded.
• Paging vs a concurrent HistoryUpdate (SqlServer). ModifiedId is an IDENTITY, which orders by allocation, not by commit. A modified read paging at the same moment a HistoryUpdate commits a modification of the same source timestamp can have already passed that row's id and so never see it on this pass (a fresh read does). The cursor is exact for a quiescent trail, not a snapshot guarantee.
• QuestDB ModifiedId across nodes. Ids are UTC ticks bumped past the last one issued by that process, so a pair's ids interleave by each node's wall clock. Same-source-time modifications made on two nodes order by their clocks (clock skew can reorder them); within one node the order is strict.
• Retention. The trail ages out on the same source-time cutoff as the samples it describes: SqlServer purges hist.SampleModified per tag alongside hist.Sample; QuestDB puts the same TTL on {p}sample_modified.
• SqlServer schema v1 (#674). With AutoCreateSchema=false a database still at schema version 1 (script 002_sample_modified.sql not applied) keeps historizing — live writes, raw/processed/at-time reads, alarm events and retention all work — but has no trail: modified reads and every HistoryUpdate request answer a node-level BadHistoryOperationUnsupported (all three arms alike — the updater throws NotSupportedException, as the read side does) with nothing sent to SQL, retention skips SampleModified, and the node logs one Warning per boot. ⚠️ Historized nodes still advertise the HistoryWrite AccessLevel bit on a v1 database: the bit is decided at materialise time from "an updater is wired", and whether the schema has the trail is only known after the backend's lazy first-use bootstrap (I/O the node manager does not do while materialising), so a v1 node is admitted to HistoryUpdate and then refused. Apply 002_sample_modified.sql and restart the node to enable them (the verdict is taken once per process). AutoCreateSchema=true (the default) still upgrades v1 to v2 in place. See Schema.
• QuestDB retry duplicates (#674). {p}sample_modified is append-only with no DEDUP key (a key would collapse genuinely distinct modifications of one sample), and each trail row gets a fresh modified_id. So a HistoryUpdate a client retries after a timeout — when the first attempt had in fact landed — records a second trail row for the same change (and, for Insert, the retry is BadEntryExists). The sample itself is idempotent (dedup upsert); only the audit trail can show the duplicate. SqlServer has the same exposure only if the client retries a committed request; there is no request id to dedupe on in either backend.
• Cross-node ordering (QuestDB). The per-tag lock is per process; the Primary-only HistoryUpdate gate (#674) is what keeps two nodes of a pair from recording concurrent updates of one tag (and so a wrong prior value). Two separate pairs sharing one QuestDB can still race; that is unsupported.
• ⚠️ Who can read the trail (#674.4). Modified reads are ordinary HistoryReads: with Security:Acl:Enabled=false (the default) any admitted session, including Anonymous, can read who rewrote history (UserName is the principal). Turn the ACL flag on, or set OpcUa:AllowAnonymous=false, if the principal names are sensitive (see security.md).
• Not implemented (#674.5): IsDeleteModified (deleting trail entries — BadHistoryOperationUnsupported); recording an Update (upsert) as a split Insert / Replace — it is recorded as Update either way; and AuditHistoryUpdateEvent (no audit event is emitted for a history change — the trail is the only record).

otopcua-cli historyread -u opc.tcp://localhost:4840 -U apikey -P otua_<keyId>_<secret> \
  -n "ns=2;s=Plant/Modbus/dev1/Speed" --start 2026-01-15T00:00:00Z --end 2026-01-16T00:00:00Z --modified

Total aggregate

The OPC UA Total aggregate is supported over the HistorianGateway backend. The gateway exposes a native Integral retrieval mode, so Total maps straight to it (HistoryAggregateType.Total → RetrievalMode.Integral) — no client-side scaling. (This replaces the retired Wonderware path, which had no Total column and derived it client-side as time-weighted Average × interval-seconds.) Bucket status codes and timestamps are preserved unchanged; empty / null buckets surface as BadNoData.

Integral unit (#677). AVEVA's integral is ∫ value dt (seconds) ÷ the tag's IntegralDivisor. The proto3 double default for HistorianTagDefinition.IntegralDivisor is 0, so GatewayTagProvisioner now sends IntegralDivisor = 1 explicitly on every EnsureTags — the value·seconds unit the SQL backends' Total uses. A tag provisioned earlier keeps its divisor until its next EnsureTags (an upsert). The live magnitude check against AVEVA (a constant 1 integrated over 1 s reads 1) is still open — it needs the VPN. Reverse processed reads on the Gateway backend are answered by AVEVA and are not re-anchored by #677.

Aggregate set and BadAggregateNotSupported (#285)

⚠️ #679 (client-visible): on the Gateway backend, Average and Count are now BadAggregateNotSupported. AVEVA's 15 retrieval modes contain no arithmetic mean and no sample count. Average used to be mapped to TimeWeightedAverage (a different statistic from the OPC UA mean the SQL backends return); Count to Counter, which is a rollover-counter increase, not a sample count. Request TimeAverage for the time-weighted number (now mapped to TimeWeightedAverage). There is no Gateway sample count. TimeAverage magnitude and timestamping against live AVEVA are unverified (needs the VPN).

HistoryAggregateType carries the wider Part 13 set (Interpolative, TimeAverage, Start, End, Delta, Range, Min/MaxActualTime, Start/EndBound, DeltaBounds, DurationGood/Bad, PercentGood/Bad, WorstQuality, NumberOfTransitions, AnnotationCount, StandardDeviationPopulation, VariancePopulation); OtOpcUaNodeManager.MapAggregate maps each standard aggregate NodeId (TimeAverage2 and Total2 were appended for #671). The SqlServer and QuestDb backends serve the seventeen listed in the aggregate table; everything else (e.g. StartBound, DurationGood, StandardDeviationPopulation) is BadAggregateNotSupported for that node. The HistorianGateway backend only has retrieval modes for the four (TimeAverage, Minimum, Maximum, Total): for any other member AggregateModeMapper throws HistoryAggregateNotSupportedException, which the HistoryRead path turns into BadAggregateNotSupported for that node only (not a generic failure, and not counted as a backend fault). The OPC UA Client driver forwards every member upstream. A NodeId outside the mapped set (e.g. StandardDeviationSample) is still BadAggregateNotSupported.

Known limitations

• Processed and AtTime are single-shot (no continuation points). Unlike Raw, neither ReadProcessedDetails nor ReadAtTimeDetails carries a client count cap (NumValuesPerNode): the Processed bucket count is deterministic (window / interval) and AtTime returns exactly one sample per requested timestamp, so the single-shot backend returns the complete result in one read and there is no "full page ⇒ maybe more" signal to page on. Returning the full result with no continuation point is spec-conformant.
• No modified-value history on the Gateway backend. Modified reads (IsReadModified = true) are served by SqlServer and QuestDb (Modified-value history, #668); the HistorianGateway exposes no modified/edited-history surface, so on Gateway (and with no historian) they return BadHistoryOperationUnsupported.

Redundancy and authorization

History reads are served from any node — there is no Primary gate. The HistoryRead AccessLevel bit set at materialization admits a node to the read dispatch. With Security:Acl:Enabled=false (the default) HistoryRead is ungated for any admitted session; with it true, the scoped HistoryRead grant decides each node (see security.md). HistoryUpdate is always gated — see HistoryUpdate.

---

---

SQL Server backend

Selected with ServerHistorian:Backend=SqlServer. Implemented in src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer (the project does not reference the Gateway project). All five seams share one connection helper, one schema bootstrapper and one tag catalog, so the schema is ensured once per process and tag ids are cached once.

Schema

Created by the embedded idempotent scripts Schema/001_initial.sql and Schema/002_sample_modified.sql (mirrored at docs/historian/sqlserver/001_initial.sql and docs/historian/sqlserver/002_sample_modified.sql); $(Schema) is the configured schema. The bootstrapper applies every script above the recorded version, in order, under the schema applock.

Object	Purpose
SchemaVersion(Version, AppliedUtc)	Applied-script ledger. Required version: 1; version 2 adds the modification trail (#668). A v1 database is served with the trail features off (#674 — see below).
Tag(TagId int identity, Name nvarchar(400) unique, DataType tinyint, EngineeringUnit, Description, CreatedUtc, UpdatedUtc)	One row per historian tagname. DataType is a DriverDataType.
Sample(TagId, SourceTimeUtc datetime2(7), ValueKind tinyint, NumValue float, IntValue bigint, StrValue nvarchar(max), StatusCode int, ServerTimeUtc)	PK (TagId, SourceTimeUtc) — one value per timestamp (Part 11), which makes redelivery idempotent and rules out tie clusters. StatusCode is the uint status stored bit-for-bit.
Event(EventId nvarchar(64) PK nonclustered, SourceName, AlarmId, AlarmName, AlarmTypeName, Severity, EventKind, Message, UserName, Comment, EventTimeUtc, ReceivedTimeUtc)	Alarm history. Clustered on (EventTimeUtc, EventId); extra index (SourceName, EventTimeUtc).
SampleModified(TagId, SourceTimeUtc, ModifiedId bigint identity, ModificationTimeUtc, UpdateType tinyint, Principal nvarchar(200), ApiKeyId, ValueKind, NumValue, IntValue, StrValue, StatusCode, ServerTimeUtc) (v2)	The Part 11 modification trail: one row per sample a HistoryUpdate changed, carrying the prior value columns (all NULL for an insert). Clustered PK (TagId, SourceTimeUtc, ModifiedId). ⚠️ No FOREIGN KEY, CHECK or trigger — it is an OUTPUT … INTO target, which forbids them; adding one fails every HistoryUpdate.
SampleRow, TimeRow table types	TVPs for batched sample writes/merges, AtTime and delete-at-time.

Column mapping per ValueKind: Double → NumValue; Int64 → IntValue + NumValue (as a double, for aggregates); Boolean → IntValue 0/1 + NumValue 0/1; String → StrValue; DateTime → IntValue (UTC ticks); Empty → all NULL. ValueKind codes: Empty 0, Double 1, Int64 2, Boolean 3, String 4, DateTime 5. Timestamps are bound as datetime2(7) everywhere (full 100 ns precision — never a DataTable, which would round to 3.33 ms).

Applying the scripts by hand (needed when AutoCreateSchema=false): create the database, then run each script in order — sqlcmd -S <server> -U <user> -P <pw> -d <db> -I -v Schema=hist -i docs/historian/sqlserver/001_initial.sql, then the same with 002_sample_modified.sql (-v Schema= supplies the $(Schema) placeholder; verified against the docker-dev SQL container). Re-running is a no-op.

⚠️ Upgrading an existing v1 historian (#668, #674). With AutoCreateSchema=true (the default) the first historian call after the upgrade applies 002_sample_modified.sql in place (it only creates SampleModified; existing samples are untouched). With AutoCreateSchema=false a v1 database is accepted (#674 — between #668 and #674 it refused every historian call, continuous-historization drains included): writes, reads, events and retention work, while modified-value reads and HistoryUpdate report BadHistoryOperationUnsupported without touching SQL, retention skips SampleModified, and one Warning per boot names the missing script. Run 002_sample_modified.sql by hand as above and restart the node to enable them (the version is read once per process). Only a schema below version 1 (missing version 1 (found 0)) still fails every call. Modifications made before the upgrade have no trail; the trail starts with the first HistoryUpdate after it.

Writes

• Live values (IHistorianValueWriter): resolve TagId through a cache, creating the Tag row on a miss (MERGE … WITH (HOLDLOCK); DataType from the first non-Empty value's kind, provisioning corrects it later), then INSERT … SELECT … WHERE NOT EXISTS through the SampleRow TVP. A null timestamp is stamped now; ServerTimeUtc = SYSUTCDATETIME(). It returns false (never throws) on a SQL failure so the recorder's retry/quarantine owns it. Semantics: delivery semantics.
• Alarm events (IAlarmHistorianWriter): per event, insert-if-absent keyed by a deterministic EventId = lowercase-hex SHA-256 over the length-prefixed fields EquipmentPath, AlarmId, EventKind, TimestampUtc.Ticks, User, Comment, Message. Outcomes: Ack on insert or already-present (including a concurrent writer winning a 2627/2601 race); RetryPlease on transient SQL errors (connection, timeout, deadlock 1205 …), bootstrap failures, cancellation and any unclassified error; PermanentFail (dead-letter) only for data errors SQL will reject forever (truncation 2628/8152, conversion, null, range, constraint). SourceName is the event's EquipmentPath; Severity is the numeric Part 9 value (Low 200, Medium 500, High 700, Critical 900).
• Tag names are nvarchar(400), and a longer name is refused, never truncated (a truncating parameter would resolve it onto whichever tag owns its first 400 characters): EnsureTags fails it individually, the live-value writer returns false for that tag (a per-tag failure, so the #538 quarantine eventually abandons it — logged as a Warning), the updater answers BadInvalidArgument for every item, and reads treat it as not found (Raw empty, AtTime / Processed BadNoData). An event read filtered by a source name over 400 characters returns no events. Keep historian tagnames under 400 characters.
• Schema bootstrap failures are cached for 5 s. With AutoCreateSchema=false and the schema missing (or SQL down), every historian call fails fast with the last bootstrap error for 5 s before SQL is tried again, so a missing schema does not serialise every write, read and alarm event behind one round-trip each; each retry that fails logs a Warning. If sp_releaseapplock fails after an auto-create, the connection pool is cleared so the session holding the applock is closed rather than returned to the pool.

Raw reads

ReadRawAsync returns every sample as stored — outage markers (Empty + Bad status) included. The one exception is defensive: a row with no value whose stored status is not Bad (only a build from before the writer forced Empty samples Bad could have written one) is served with BadNoData, on Raw and AtTime alike.

Request	Window	Order
start < end	[start, end)	ascending
end < start (Part 11 reverse)	(end, start]	descending
start == end	the sample exactly at that time	—

NumValuesPerNode == 0 is unbounded (larger values clamp to int.MaxValue). An unknown tag is an empty result (GoodNoData); reads never create tag rows. The node manager's paging is unchanged; because the PK rules out tie clusters, resuming "from the last timestamp inclusive and trimming it" is exact.

AtTime reads

One statement per call: the distinct requested times go in a TimeRow TVP and each picks up its previous-or-equal sample p and next sample n via two OUTER APPLYs. A usable sample has severity Good or Uncertain and a value (an Empty sample is an outage marker and counts as Bad whatever its code). Per requested time T (the result is stamped T):

Case	Result
p at exactly T	the stored value and status, unchanged
no p (before the first sample), or unknown tag	BadNoData, null
p is Bad (T lies inside an outage)	null value carrying p's Bad status code (info bits stripped) — never better than its source
p usable, n Bad	p's value held (not interpolated into the outage), UncertainDataSubNormal + Interpolated
Double / Int64 with a usable n	linear interpolation of NumValue; Int64 is computed through double and rounded to the nearest integer (ties away from zero), keeping the Int64 type. Status Good + Interpolated when both neighbours are Good, else UncertainDataSubNormal + Interpolated
Boolean / String / DateTime, or T after the last sample	stepped: p's value with p's status + Interpolated
T after the last sample and after now	p's value held, UncertainDataSubNormal + Interpolated (the value is not known to still hold)

Setting Interpolated (or, in aggregates, Calculated) first clears any historian bit the stored status already carries, so a result never has both.

⚠️ Because Int64 interpolation goes through double, magnitudes beyond 2^53 lose precision.

Events

[start, end) by EventTimeUtc; through the five-argument read a reversed window (end < start) is swapped and returned ascending. The direction-aware overload with reverse: true (what the node manager calls for a reverse read, #675) reads (lo, hi] ordered (EventTimeUtc, EventId) descending — the clustered index scanned backwards. A null sourceName means all sources, otherwise SourceName = @source. Forward is ordered (EventTimeUtc, EventId) ascending. A bounded read fetches maxEvents + 1 rows and returns a non-null continuation point when the extra row exists; the node manager turns that flag into an OPC UA continuation point (see Continuation-point paging (Events)).

Aggregates (Processed reads)

Two kinds. The per-sample aggregates (Average, Minimum, Maximum, Count, Range, Start/End/Delta, Min/MaxActualTime, PercentGood/Bad) are simple per-sample definitions — no time-weighting, no interval-boundary values — pushed down to SQL. The time-weighted aggregates (TimeAverage, TimeAverage2, Total, Total2, Interpolative; #671) follow OPC UA Part 13 with bounding values — see Time-weighted aggregates. Any HistoryAggregateType not in the tables throws HistoryAggregateNotSupportedException ⇒ BadAggregateNotSupported for that node (e.g. StartBound, DurationGood, StandardDeviationPopulation).

⚠️ Release note / migration (#671): Total changed meaning on the SqlServer and QuestDb backends. It was the plain SUM of the usable samples in the interval, pushed down to SQL; it is now the Part 13 time-weighted integral, TimeAverage × interval seconds (value·seconds), evaluated in C# from the raw rows (not pushed down) and therefore subject to ServerHistorian:MaxTimeWeightedRawRows — a long Total over a dense tag that used to succeed may now fail BadTooManyOperations; read it in shorter windows. This brings Total closer to the Gateway backend, whose Total maps to AVEVA's native time-weighted Integral retrieval mode — but AVEVA's integral time unit/divisor has not been verified, so the two backends may differ by a constant factor for the same data. A client that relied on the old sum gets a different number for the same data (e.g. two samples 10 and 20 ten seconds apart no longer total 30). There is no switch back; Count × Average reproduces the old sum.

⚠️ Release note (#677): reverse processed reads changed — client-visible. A reverse read (end < start) is now anchored at the request's start (its later time) for every aggregate, as OPC UA Part 13 and the SDK's own AggregateCalculator do: intervals step back from start, the window is (end, start], the truncated interval is the earliest one, and each result is stamped with its interval's later edge. Before #677 only Interpolative did this; every other aggregate swapped the window, anchored it at the earlier time ([end, start), the partial interval at the latest end) and stamped each interval with its earlier edge. So a reverse 80 s → 0 s read at 20 s used to answer [60,80) [40,60) [20,40) [0,20) stamped 60/40/20/0; it now answers (60,80] (40,60] (20,40] (0,20] stamped 80/60/40/20 — a sample exactly on an interval edge now belongs to the interval it ends, a sample at start is included and one at end is not (the same window a reverse raw read uses). Forward reads are unchanged apart from the new (forward-only) start-of-data Partial bit below. A client that paired reverse results with forward ones by timestamp must shift by one interval.

Bucketing. Forward: one result per interval of [start, end); bucket i = [start + i·interval, start + (i+1)·interval), start-inclusive / end-exclusive to the tick, the last bucket truncated at end, each stamped with its start. Reverse (end < start, #677): the window is (end, start]; result k = (start − (k+1)·interval, start − k·interval], clipped at end (the earliest interval is the truncated one), stamped with its later edge start − k·interval, returned latest first. interval <= 0 ⇒ a single interval [start, end); start == end ⇒ no intervals. The bucket index is computed in 100 ns ticks (not milliseconds): DATEDIFF_BIG(microsecond, @start, t) × 10 plus a sub-microsecond correction from the two tick digits, divided by the interval in ticks. (DATEDIFF counts unit boundaries crossed, so a millisecond formulation misplaces samples near a bucket edge, and a nanosecond one overflows past ≈292 years.) A reverse read adds one constant to that elapsed-ticks dividend (BucketPlan.BucketIndexAdjustTicks; QuestDB folds it into its base) that re-anchors the grid at the later time and makes the buckets later-edge-inclusive — BucketPlan.IndexOf is the C# statement of the math both dialects mirror. A window of more than 100 000 intervals is refused before any I/O and reported BadHistoryOperationUnsupported for that node. Every returned status carries the historian Calculated info bit, BadNoData included.

Partial bit (#677). On a forward read, every aggregate except plain Average sets the Partial bit (0x4) on a non-Bad result for the interval the tag's data starts inside — its earliest stored sample (any quality) lies strictly between the interval's edges, so there is no bounding value before it and the data does not cover the whole interval. This follows the SDK's m_startOfData rule, which the SDK turns off for Average and in effect applies to forward reads only (a reverse read is fed latest-first, so its "start of data" is the latest sample, past the window). A reverse read therefore sets no start-of-data Partial — matching the SDK. Only a forward per-sample read of an aggregate other than Average fetches the tag's earliest sample (one seek, bounded by the window end); every other read skips that query. An earliest sample exactly on an edge, or before the interval, sets nothing. The time-weighted aggregates also keep setting Partial on the truncated interval (below).

Sample classes. Usable = severity Good or Uncertain and a stored value (not Empty). Good = usable with severity Good. Bad = not usable.

Aggregate	Computed over	Value timestamp
Average, Minimum, Maximum	AVG / MIN / MAX of NumValue over usable samples	bucket start (reverse read: its later edge, as every row marked "bucket start" here)
Count	number of usable samples (Int32)	bucket start
Range	Maximum − Minimum	bucket start
Start, End	first / last usable sample by time (keeps the sample's type)	bucket start
Delta	End − Start (numeric samples)	bucket start
MinimumActualTime, MaximumActualTime	the usable sample with the smallest / largest NumValue; ties ⇒ earliest	the sample's own time
PercentGood	100 × (Good samples) / (all samples)	bucket start
PercentBad	100 × (Bad samples) / (all samples)	bucket start

Status and edge rules:

• An empty interval (no samples) ⇒ BadNoData. Count — and every value aggregate — over an interval whose samples are all Bad ⇒ BadNoData, not 0.
• Value aggregates are Good when every sample in the bucket is Good and UncertainDataSubNormal when the bucket also holds any Bad or Uncertain sample.
• Count, Start, End, PercentGood, PercentBad accept any value kind; the other aggregates (and Delta) on a bucket that holds a usable String/DateTime sample ⇒ BadAggregateInvalidInputs for that interval.
• PercentGood / PercentBad and Uncertain samples: Uncertain samples are counted in the denominator but are neither Good nor Bad, so PercentGood + PercentBad < 100 whenever a bucket holds Uncertain samples. Both report status Good for any non-empty bucket.
• Everything except Count and Start/End is a double; Int64 samples aggregate as doubles and Boolean as 0/1.

Time-weighted aggregates (Part 13, #671)

TimeAverage, TimeAverage2, Total, Total2 and Interpolative, evaluated by the shared Driver.Historian.Sql.Common/TimeWeightedAggregator.cs for both SQL backends (so SqlServer and QuestDb agree exactly). This whole family — Total included — is evaluated in C#, not pushed down to SQL (see Pushdown trade-off and the raw-row cap below). Bucketing (edges, truncated interval, reverse anchoring, the 100 000-interval cap) is the same as above.

Server defaults, fixed: TreatUncertainAsBad = false, PercentDataBad = PercentDataGood = 100, UseSlopedExtrapolation = false (ServerAggregateConfiguration in the OPC UA server project — the SDK's own AggregateManager defaults, which a test pins). A client's AggregateConfiguration (#677): UseServerCapabilitiesDefaults = true, or the four values spelled out equal to these defaults ⇒ served; anything else ⇒ BadAggregateConfigurationRejected for each node of the request (the source is never called). It used to be silently ignored, so a client asking for, say, TreatUncertainAsBad = true got answers computed without it. The check is server-wide (it applies on the Gateway backend too). A client sending an all-zero structure without UseServerCapabilitiesDefaults is asking for PercentDataBad = PercentDataGood = 0 and is refused; the SDK's default-constructed AggregateConfiguration sets UseServerCapabilitiesDefaults = true and is served.

Bounding values. Besides the window [start, end), each read fetches four single-row seeks: the latest usable sample before start, the latest sample of any quality before start (they differ when an outage straddles the start), the first sample of any quality at or after end, and the first usable one at or after end. So a bucket with no samples of its own, between two samples, still gets a value.

The signal. Between a usable sample p and the next sample s: both Double/Int64 ⇒ sloped (linear), segment quality the worse of the two; either Boolean ⇒ stepped (p held), quality p's; s Bad ⇒ p held up to the outage (never interpolated into it — the AtTime rule), Uncertain; no later sample at all ⇒ p held to the window end (stepped extrapolation), Uncertain. From a Bad sample (incl. an Empty outage marker) to the next sample the signal is Bad (outage); before the first sample there is no data. That is the simple-bounding signal. The interpolated-bounding signal skips Bad samples: it joins consecutive usable samples (sloped / stepped) straight across an outage.

Aggregate	Value	Bad intervals
TimeAverage	∫ interpolated-bounding signal ÷ its defined duration	interpolated across (Part 13)
Total	TimeAverage × interval seconds (value·s; the truncated last bucket uses its own length)	interpolated across
TimeAverage2	∫ simple-bounding signal over non-Bad time ÷ non-Bad duration	excluded
Total2	∫ simple-bounding signal over non-Bad time (value·s)	excluded
Interpolative	interpolated-bounding value at the interval start (reverse read: see below)	interpolated across

Status, the four time-weighted values — aligned with the OPC Foundation SDK's AggregateCalculator.GetTimeBasedStatusCode (inspected in Opc.Ua.Server 1.5.378.106) under the fixed defaults. With TreatUncertainAsBad = false, Uncertain time counts as Good time: Uncertain samples, a value held up to an outage and the stepped extrapolation past the last sample all count toward PercentDataGood.

• Every instant of the interval non-Bad ⇒ Good.
• Some non-Bad time but not all (an outage, or time before the first sample) ⇒ UncertainDataSubNormal.
• No non-Bad time but some outage time ⇒ generic Bad (0x80000000) — the SDK's all-Bad result, value null.
• No data at all ⇒ BadNoData.
• A usable String/DateTime sample feeding the interval ⇒ BadAggregateInvalidInputs.

All carry the Calculated bit; a non-Bad result for the truncated interval (shorter than the processing interval — the latest one forward, the earliest one reverse) or for the interval the tag's data starts inside (#677, see Partial bit above) also carries the Partial bit (0x4, AggregateBits.Partial). Timestamp = interval start (reverse: its later edge); value double (Boolean 0/1).

A live tag's latest interval. Past the tag's last stored sample the value is held (stepped extrapolation, Uncertain). Under the SDK rule that Uncertain time is Good time, so the latest interval of a live tag normally reads Good — not Uncertain — with the last value held to the interval end. It reads Uncertain only if the interval also contains outage time or starts before the tag's first sample. Do not take a Good latest interval as proof that data arrived up to its end. If it was truncated by the request's end time it also carries Partial.

Status, Interpolative (point-based, not the PercentDataGood rule):

• A usable sample exactly at the point ⇒ that value with its own stored status (no historian bit).
• No usable sample at or before the point ⇒ plain BadNoData.
• Otherwise the value with the Interpolated bit:
  • Good on a segment whose defining samples are Good;
  • UncertainDataSubNormal inside or up to an outage (bridged) and past the last sample (extrapolated).

Reverse reads (end < start). Part 13 anchors a reverse read at the request's start, which is the later time — for every aggregate since #677. Interpolative's points are start, start − interval, …, latest first: a reverse read 10:00 → 09:00 at 1 h returns the value at 10:00, not 09:00, and no point ever falls before the window. The interval aggregates use the intervals (start − (k+1)·interval, start − k·interval], the earliest one truncated, each stamped with its later edge.

SDK parity (#677). SdkAggregateParityTests (unit, Sql.Common.Tests) and ProcessedConformance.Processed_reads_match_the_SDK_aggregate_calculators (both SQL backends, through the real SQL) run the pinned SDK's own calculators (AverageAggregateCalculator, MinMaxAggregateCalculator, CountAggregateCalculator, the base AggregateCalculator for Interpolative) over the same samples, forward and reverse, and compare every interval's timestamp, value, status code and Partial bit. The per-sample comparison covers Average, Minimum, Maximum, Count, Range, Start, End, Delta, Minimum/MaximumActualTime and PercentGood/Bad; reverse reads over data that starts inside the window are compared too. Known differences that remain (tracked as #680) — documented and masked in the tests, deliberately not changed here because forward results were to stay as they are — are: Start/End (the SDK returns the raw first/last sample, of any quality, with its own timestamp and stored status; ours returns the first/last usable value stamped with the interval, under the per-sample status rule — the values agree wherever the SDK's is not Bad); Delta status (the SDK's stays Good with a Bad sample in the interval, ours is Uncertain); PercentGood/PercentBad (ours count samples, the SDK measures duration — different definitions, so only timestamp and status are compared); the SDK reports TimeAverage/Total Uncertain whenever any region of the interval is non-Good, where ours applies the time-based rule (a fully-covered interval bridged across a Bad sample is Good here); Total over an interval the data starts inside is TimeAverage × interval here but the integral of the covered part in the SDK; ours flags a truncated interval Partial, the SDK does not; the SDK's forward end-of-data Partial rule is not implemented; the SDK's per-sample Count answers 0/Good for an empty interval inside the data and its per-sample aggregates use Good samples only; and on a reverse Interpolative read the SDK marks a sample exactly at a point Interpolated (Uncertain if a Bad sample follows) where ours — like the SDK forward — returns the raw sample.

Pushdown trade-off. Unlike the per-sample aggregates, the time-weighted family is not pushed down to SQL: the backend streams the window's rows plus the four bounds (one batch of five result sets on SQL Server; a bounds UNION ALL + a window select on QuestDB) and the evaluator splits segments at bucket edges in C#. Network traffic is therefore O(samples in the window) rather than O(buckets); evaluation memory stays O(buckets) (rows are streamed, never materialised). The alternative — the same Part 13 segment interpolation, edge splitting and two bounding definitions written twice, in T-SQL and QuestDB SQL, and kept bit-identical — was judged the larger risk. For very long, densely sampled windows prefer the pushed-down Average (or a coarser read) over TimeAverage.

Raw-row cap. The window query is limited to ServerHistorian:MaxTimeWeightedRawRows + 1 rows (default 1 000 000, shared by both SQL backends; TOP (@maxRows) / LIMIT $4). When that extra row arrives the read stops and the node fails with BadTooManyOperations — for that node only; it is not counted in the backend's health failures. Without the cap a one-year Total on a 1 Hz tag (≈31 M rows) would hold a pooled connection until HistoryReadDeadline and then return BadTimeout. The four bound rows do not count toward the cap, and the per-sample aggregates are never capped. Raise the cap, or read in shorter windows.

Health

The read client reports GetHealthSnapshot() (query/failure counters, consecutive failures, LastError — e.g. a missing schema under AutoCreateSchema=false). A SQL error on a read surfaces as BadHistoryOperationUnsupported for that node only; a cancelled read (deadline, client abort) is not counted as a backend fault.

Retention (RetentionDays)

hist.Sample, hist.SampleModified and hist.Event grow without bound unless ServerHistorian:SqlServer:RetentionDays is set. QuestDB already has an equivalent TTL (QuestDb:RetentionDays); the Gateway backend's retention is the AVEVA Historian's own.

• What runs. HistorianRetentionService (a hosted service, driver-role nodes, Backend=SqlServer with RetentionDays set; inert otherwise) ticks every PurgeInterval (first pass at start-up) and calls the backend's IHistorianRetention.PurgeAsync.
• Gate vs. mutual exclusion. Each tick consults IHistorizationRoleView.ShouldRecordHistorizedValues (the capture verdict; a standby skips, an unknown role runs). That verdict is pair-scoped while the purge is database-scoped, so the gate is only a load reducer: the sp_getapplock below is the actual mutual exclusion. Pairs sharing one historian database, and both nodes of a pair on their boot-time pass, simply serialise on it. Set RetentionDays uniformly on every node that shares a historian database — if they differ, the smallest window wins.
• How it deletes. Cutoff = UtcNow - RetentionDays on source time. Samples are purged per tag with DELETE TOP (@n) ... WHERE TagId = @id AND SourceTimeUtc < @cutoff (a seek on the clustered (TagId, SourceTimeUtc) key; a global time predicate would scan) — and, in the same per-tag pass, the modification trail with DELETE TOP (@n) FROM SampleModified WHERE TagId = @id AND SourceTimeUtc < @cutoff (#668; same source-time cutoff, clustered seek; the purge itself never writes a trail row) — then events with DELETE TOP (@n) ... WHERE EventTimeUtc < @cutoff (seek on CX_Event_Time), each looped until a batch comes back short. Cancellation is honoured between batches. The hist.Tag catalogue is never purged.
• Shared databases. Several pairs may point at one database; a session sp_getapplock (OtOpcUa.Historian.Retention.<schema>, timeout 0) lets one purger run and the others skip that tick (Debug log).
• Observability. Counter otopcua.historian.retention.purged{table=Sample|SampleModified|Event}; a failed pass logs a Warning and retries next interval — it never affects capture or reads.
• Schema v1 (#674). On a v1 schema (no SampleModified, AutoCreateSchema=false) the trail purge is skipped and omitted from the pass results; samples and events purge normally.
• Not done: partition switching. For very large installs (billions of rows) a partitioned hist.Sample with sliding-window SWITCH/drop is cheaper than row deletes; the purger is deliberately simple batched deletes. Another table is added by one more DeleteLoopAsync call in SqlServerHistorianRetention.PurgeAsync (as hist.SampleModified was, #668).

Docker-dev recipe

The docker-dev sql container (otopcua-dev-sql-1, host port 14330, sa with the committed dev password from docker-dev/docker-compose.yml) already hosts the ConfigDb; give the historian its own database on the same instance.

# 1. Create the database (the backend creates the schema + tables on first use, not the database).
sqlcmd -S localhost,14330 -U sa -P 'OtOpcUa!Dev123' -C -I -Q "CREATE DATABASE OtOpcUaHistorian"

# 2. Point the central nodes at it. Add to the `environment:` of central-1 AND central-2 in docker-dev/docker-compose.yml
#    (or a compose override), rebuild the images (the rig image goes stale), then `up -d`:
#      ServerHistorian__Enabled: "true"
#      ServerHistorian__Backend: "SqlServer"
#      ServerHistorian__SqlServer__ConnectionString: "Server=sql,1433;Database=OtOpcUaHistorian;User Id=sa;Password=OtOpcUa!Dev123;TrustServerCertificate=True"
#      ContinuousHistorization__Enabled: "true"
#      ContinuousHistorization__OutboxPath: "/app/data/historization-outbox"
#    (ServerHistorian__Endpoint / __ApiKey are not used by this backend.)

# 3. Mark a rig tag historized ("isHistorized": true in its TagConfig), deploy, then watch rows land:
sqlcmd -S localhost,14330 -U sa -P 'OtOpcUa!Dev123' -C -I -d OtOpcUaHistorian \
  -Q "SELECT t.Name, COUNT(*) n, MAX(s.SourceTimeUtc) last FROM hist.Sample s JOIN hist.Tag t ON t.TagId=s.TagId GROUP BY t.Name"

Only the Primary of a pair records, so the row count advances once per published value, not twice. To watch retention, add ServerHistorian__SqlServer__RetentionDays: "1" and ServerHistorian__SqlServer__PurgeInterval: "00:00:30", insert a hist.Sample row with an old SourceTimeUtc for an existing tag, and see it vanish within a tick while recent rows stay. Read it back with Client.CLI historyread and rewrite it with Client.CLI historyupdate (Client.CLI.md). The password above is the committed docker-dev credential, not a secret; use ${secret:…} / environment for anything real.

Integration tests

tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests (bootstrap incl. two concurrent bootstrappers, every kind round-trip, redelivery, raw / AtTime / aggregate / event reads, every update result code) is gated on the environment variable HISTORIAN_SQL_TEST_CONNECTION — a connection string with CREATE/DROP DATABASE rights. Absent ⇒ every test skips. Present ⇒ each run creates a throwaway OtOpcUaHistorian_T_<guid> database and drops it on dispose.

export HISTORIAN_SQL_TEST_CONNECTION='Server=localhost,14330;User Id=sa;Password=OtOpcUa!Dev123;TrustServerCertificate=True;Encrypt=False'
dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests

---

QuestDB backend

Selected with ServerHistorian:Backend=QuestDb. Implemented in src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb (pure logic shared with the SQL Server backend lives in Driver.Historian.Sql.Common). All access is the Postgres wire protocol via Npgsql (no ILP, no extra package). Design: plans/2026-10-10-questdb-historian-design.md; the spike that pinned every QuestDB-specific form: plans/2026-10-10-questdb-spike.md. Pinned image: questdb/questdb:10.0.1.

QuestDB serves every value kind (Double, Int64, Boolean, String, DateTime, Empty), stores the real status, records comms-loss outage markers, serves the seventeen aggregates (time-weighted ones through the same shared evaluator, so results are identical) and supports HistoryUpdate (soft delete), exactly like SqlServer. The same conformance suite (tests/Drivers/Historian.Conformance) runs against both backends; the differences below are the only ones.

Configuration (ServerHistorian:QuestDb)

Read only when Backend=QuestDb; bound by HistorianBackendRegistry.BindQuestDb, validated (consumer-gated, like SqlServer) by QuestDbHistorianOptions.Validate(). The gateway keys (Endpoint, ApiKey, UseTls) are not required.

Key	Default	Validation	Meaning
QuestDb:ConnectionString	""	Required (non-blank)	Npgsql connection string for QuestDB's PG-wire port, e.g. Host=questdb;Port=8812;Username=admin;Password=quest;Database=qdb;ServerCompatibilityMode=NoTypeLoading. ServerCompatibilityMode=NoTypeLoading skips the pg_type load; every query form was verified with it. Supply via env ServerHistorian__QuestDb__ConnectionString / ${secret:} for anything real.
QuestDb:TablePrefix	hist_	^[A-Za-z0-9_]+$	QuestDB has no schemas, so the four tables are {prefix}sample, {prefix}event, {prefix}tag and {prefix}sample_modified (#668). Spliced into DDL/DML, hence the regex.
QuestDb:RetentionDays	unset	>= 1 when set	Applied as table TTL to {prefix}sample, {prefix}event and {prefix}sample_modified (see Retention). Unset keeps data forever.
QuestDb:CommandTimeout	00:00:30	> 0	Per-command timeout, rounded up to whole seconds (minimum 1 s).
QuestDb:AutoCreateSchema	true	—	true: create the tables (and reconcile the TTL) on first use. false: only verify the four tables exist (⚠️ an install from before #668 lacks {prefix}sample_modified — create it with the DDL in QuestDbTables.CreateSampleModified before upgrading, or every call fails); if one is missing every historian call fails with HistorianSchemaMissingException (health LastError, no crash) and no TTL is touched. A failed bootstrap is not latched: it is re-thrown for 5 s without touching QuestDB, then retried.
QuestDb:VisibilityTimeout	00:00:05	>= 0	How long a write waits for its rows to become visible to queries (WAL visibility). 0 disables the wait.

⚠️ Connection pool size. QuestDB's default limit is 64 concurrent PG-wire connections (pg.net.connection.limit), while Npgsql's own default Maximum Pool Size is 100. A busy node (HistoryRead batches, the recorder, the updater and alarm writes all share one pool) could therefore be refused by QuestDB before Npgsql's pool is full. When the connection string does not set Maximum Pool Size (or MaxPoolSize), the backend now caps it at 24, so the two nodes of a redundant pair sharing one QuestDB stay under the server limit together (2 × 24 = 48). An explicit value in the connection string is always kept as written — including one above 64, since QuestDB's limit is server-configurable. The effective value is logged once at Information when the backend first builds its pool.

Tables

All are WAL tables; the first three carry DEDUP UPSERT KEYS, so re-sending a row with the same key replaces it (the upsert is what makes HistoryUpdate Replace work; the writer's own insert-if-absent semantics are built on top, below). The modification trail is append-only.

Table	Key	Partitioning
{p}sample (ts, tag, kind, num_value, int_value, str_value, status, deleted, server_ts)	(ts, tag)	PARTITION BY DAY (TTL when configured)
{p}event (one-for-one with hist.Event)	(ts, event_id)	PARTITION BY DAY (TTL when configured)
{p}tag (ts, name, data_type, eng_unit, description, updated_utc)	(ts, name) with ts fixed at 1970-01-01 — an upsert by name	PARTITION BY YEAR, never expires
{p}sample_modified (ts = the sample's source time, tag, modified_id, modification_ts, update_type, principal, api_key_id, kind, num_value, int_value, str_value, status, server_ts) (#668)	none — append-only, no DEDUP; modified_id (process-monotonic ≈ UTC ticks) orders a timestamp's trail	PARTITION BY DAY (TTL when configured). A NULL status marks "no prior value" (QuestDB BYTE has no NULL, so kind is 0)

status holds the uint OPC UA status code unsigned in a LONG (so "Bad" is status >= 2147483648, not < 0); int_value also holds Booleans as 0/1; kind is the HistorianValueKind in a BYTE. Tag names are limited to 400 characters (longer is a per-tag failure, never truncated). {p}tag is the only "is this tag known" authority (QuestDbTagCatalog): reads never create a tag, Insert/Update and the live writer do, Replace and the deletes do not.

Timestamps: microseconds

QuestDB's designated timestamp is a microsecond. Every inbound DateTime is normalised by QuestDbConnection.Round (Unspecified ⇒ UTC, Local ⇒ UTC, ticks truncated to a multiple of 10, MinValue/MaxValue nudged inward because Npgsql would send them as ±infinity) and bound only through BindTimestamp. Consequences:

• Two live samples for one tag less than 1 µs apart collapse to one; the first wins. A batch is deduplicated client-side after rounding.
• Raw and Processed results carry stored (µs) timestamps. A sub-µs request bound is rounded, and the rounded start is the request start, so a Processed bucket's timestamp is the rounded window start (a sub-µs label divergence from SqlServer, which labels with the exact start; values and bucket membership are identical).
• AtTime evaluates at the rounded time but stamps each result with the time the client asked for, so a client can match answers to requests. Stored data is µs-only, so a sub-µs request adjacent to a sample is an exact hit.
• IHistorianDataSource.TimestampResolutionTicks is 1 by default and 10 on QuestDB. HistoryRead-Raw paging steps off a drained timestamp by that amount (forward T + resolution, reverse T − resolution). Stepping by 1 tick would be truncated back to T and a one-value-per-page forward read would loop forever. Same-timestamp ties (legal on the Gateway backend) are still paged within the timestamp.
• Pre-1970 timestamps cannot be stored (QuestDB: designated timestamp before 1970-01-01 is not allowed; the whole INSERT fails). So:
  • the live writer drops such samples with at most one Warning per tag per 10 minutes (a device with an unset clock would otherwise warn every drain pass; every dropped sample is still counted in DroppedPre1970) — a retry could never succeed, and sending them would push the tag into the #538 quarantine;
  • HistoryUpdate Insert / Replace / Update of such an item returns BadOutOfRange (nothing is sent); DeleteAtTime returns BadNoEntryExists; DeleteRaw finds nothing;
  • an alarm event with such a time is a PermanentFail (dead-letter);
  • read bounds before 1970 work (a Raw/Processed window may start before the epoch; bounds are never clamped, only MinValue/MaxValue nudged).
• Int64 long.MinValue. QuestDB's LONG NULL is long.MinValue, so a stored FromInt64(long.MinValue) has int_value NULL. The read path repairs it (kind = Int64 ∧ int_value IS NULL ∧ num_value = −2^63 ⇒ long.MinValue); without the repair it would read back as Empty/0. Likewise NaN/±Infinity doubles read back as NULL, so they are stored as Empty + BadOutOfRange (live) or rejected BadInvalidArgument (HistoryUpdate), as on SqlServer.

Live writes: insert-if-absent, tombstones included

The recorder's writer (QuestDbHistorianValueWriter) has the same first-wins redelivery contract as SQL Server:

1. group by tag, round to µs, drop in-batch duplicates (first wins), drop pre-1970 samples;
2. read the timestamps that already exist for the tag (one range query per cluster of nearby times, deleted rows included);
3. INSERT only the absent rows — parameterised multi-row, at most 1,000 rows per statement;
4. wait for WAL visibility.

Because DEDUP UPSERT would otherwise let a re-sent sample overwrite the stored one, step 2 is what keeps outbox redelivery idempotent — and a live-writer redelivery never resurrects a deleted row: a tombstone counts as present. Any SQL or transport failure returns false (the recorder retries; the #538 per-tag quarantine still applies); the writer never throws.

WAL visibility and the read-your-writes wait

QuestDB applies WAL commits asynchronously, so a row just inserted may not be visible to the next query (measured: up to 7 of 20 inserts invisible to an immediate re-read). Every write path waits — the live writer, the HistoryUpdate updater (including between DeleteRaw pages), the tag catalog/provisioner and the alarm writer (QuestDbWalVisibility): after the INSERT(s) it reads sequencerTxn from wal_tables() and polls writerTxn about every 10 ms until it catches up, the table is reported suspended (Warning, stop), or VisibilityTimeout elapses (Warning). A timeout never changes a result code and never discards data; the rows simply become visible shortly after. Measured: median 0 ms, p95 6 ms.

HistoryUpdate (soft delete)

All five operations are supported, and every one is an INSERT: QuestDB UPDATE/DELETE are never issued. "Exists" means a row that is present and not deleted.

Operation	Effect	Result
Insert	write the row when none exists — a tombstone counts as absent, so Insert over a tombstone succeeds	GoodEntryInserted / BadEntryExists
Replace	overwrite (dedup upsert) when a live row exists	GoodEntryReplaced / BadNoEntryExists
Update	write either way	GoodEntryReplaced or GoodEntryInserted
DeleteAtTime	write a tombstone (deleted = true, value NULL, BadNoData) where a live row exists	Good / BadNoEntryExists
DeleteRaw	tombstone every live row in [start, end), 1,000 per page, waiting for visibility between pages	the same codes as SqlServer

Modification trail (#668). Each operation first appends one {p}sample_modified row per sample it will change, carrying the live row's current value (read by the same existence query), then writes the change. No multi-table transaction exists, so this order is the guarantee: a crash between the two leaves an orphan trail row, never an untracked change — the documented deviation from SqlServer's atomic trail. See Modified-value history.

The per-item codes come from the shared HistoryUpdatePlanner, so they equal SqlServer's (including the duplicate-timestamp-in-one-request rules above). Every read and aggregate filters NOT deleted. A client Insert at a deleted timestamp re-populates it (the OPC UA intent); only the live writer's redelivery is barred from doing so.

⚠️ Concurrency. The updater serializes calls per tag, in-process only (64 striped locks held from the existence read through the visibility wait), so two concurrent Insert/Replace/Delete batches on one tag cannot both act on a stale read. It does not lock against the live writer, nor against a second node. The existence checks read committed data, so a recorder sample at the same timestamp still in flight in the WAL can race an Insert/Replace (updates normally target past timestamps), and two nodes writing the same new key concurrently can produce last-write-wins (QuestDB has no UPDLOCK; the Primary gate makes it rare). This residual is accepted.

Reads

Every query filters NOT deleted; every value goes through StoredSample, so the quality clamp equals SqlServer's.

• Raw: ts range, ORDER BY ts [DESC] LIMIT $n (bound, int); ContinuationPoint is always null — the node manager pages by re-reading from the last timestamp ± TimestampResolutionTicks.
• AtTime: previous-or-equal and next neighbour of each time by LIMIT 1 seeks, UNION ALL-ed, 50 times per statement (QuestDB 10.0.1 refuses more than 126 branches). AtTimeEvaluator decides each value, as on SqlServer.
• Events: [start, end) by ts, ordered (ts, event_id), maxEvents + 1 fetched to produce the continuation point; a reverse read (#675) is (lo, hi] ordered (ts, event_id) descending.
• Processed: one query per node; bucket index (cast(ts AS LONG) * 10 - bucketBaseUnixTicks) / intervalTicks (never SAMPLE BY, whose alignment differs), so bucket edges are identical to SqlServer (forward start-inclusive / end-exclusive with the partial bucket last; reverse anchored at the later time, later-edge-inclusive, partial bucket earliest — #677; interval 0 ⇒ one bucket, 100,000-bucket cap). On a forward read of any aggregate but Average, a second LIMIT 1 statement (capped at the window end) fetches the tag's earliest live sample for the start-of-data Partial bit. Min/max times use row_number() windows with the earliest-timestamp tie-break.

Retention (TTL)

RetentionDays becomes TTL n DAYS on {p}sample, {p}event and {p}sample_modified (never on {p}tag: TTL must be a multiple of the partition size). QuestDB drops expired day partitions itself, tombstones included.

• At bootstrap the current TTL is read from tables() (ttlValue, ttlUnit). QuestDB normalises the unit (7 days reads 1 WEEK), so the bootstrapper compares in hours (HOUR=1, DAY=24, WEEK=168; any other unit counts as different) and issues ALTER TABLE … SET TTL n DAYS only on a difference (SET TTL 0 DAYS removes it). The ALTER is applied through the WAL, so it does not re-read to verify; a very large RetentionDays that QuestDB normalises into MONTH/YEAR is re-ALTERed on every boot (harmless).
• ⚠️ TTL is enforced only on write commits. QuestDB checks for expired partitions when a write commits to the table, so a quiet table never drops anything: partitions older than the TTL linger until the next write lands. Do not read a stale-looking partition as a broken TTL.
• Expiry is judged against the wall clock (min(max timestamp, now)), so a future-dated sample does not expire current data. ⚠️ Keep cairo.ttl.use.wall.clock at its default, true (#674). It is what QuestDB documents as protecting a TTL table from a future-dated row; with it false the TTL would be measured from the table's newest timestamp and one far-future row (a 2062 typo) could drop every older partition — including {p}sample_modified, the record of who changed history. Evidence (QuestDB 10.0.1, 2026-10-11): the default is true, and a 2066 row inserted into a 30-day-TTL table evicted nothing inside the TTL, while a partition genuinely older than 30 days was evicted by that same commit — pinned by TtlWallClockTests. (The same run with the flag false also kept the in-TTL partitions, so the test pins the outcome rather than isolating the flag; do not rely on that.) The server-side ServerHistorian:HistoryUpdateMaxFutureSkew bound (default 1 day) is the primary guard: a HistoryUpdate cannot write a far-future row.
• The TTL is visible in table metadata: SELECT table_name, ttlValue, ttlUnit FROM tables() WHERE starts_with(table_name, 'hist_').

Docker-dev recipe

QuestDB is an optional compose overlay, docker-dev/docker-compose.questdb.yml, so the default rig is unchanged. See docker-dev/README.md.

# 1. Bring up ONLY the questdb service (PG wire on host :18812, console/REST on host :19000).
docker compose -f docker-dev/docker-compose.yml -f docker-dev/docker-compose.questdb.yml up -d --no-deps questdb

# 2. Recreate the central nodes against it (the QuestDb connection string already defaults to the `questdb` service):
OTOPCUA_HISTORIAN_ENABLED=true OTOPCUA_HISTORIAN_BACKEND=QuestDb OTOPCUA_CONTHIST_ENABLED=true \
  docker compose -f docker-dev/docker-compose.yml -f docker-dev/docker-compose.questdb.yml up -d central-1 central-2

# 3. Mark a rig tag historized, deploy, then watch rows land (REST, no client needed):
curl -G localhost:19000/exec --data-urlencode "query=SELECT tag, count() n, max(ts) last FROM hist_sample WHERE NOT deleted GROUP BY tag"

Only the Primary of a pair records. Read back with Client.CLI historyread, rewrite with Client.CLI historyupdate.

Integration tests

tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests (bootstrap incl. two concurrent bootstrappers, TTL via tables(), writes, tombstones and redelivery, the visibility wait, updater, reads, paging, and the shared conformance suite) is gated on HISTORIAN_QUESTDB_TEST_CONNECTION — an Npgsql connection string for the PG-wire port. Absent ⇒ every test skips. Each test class uses a unique throwaway prefix (t_<guid>_) and drops its tables afterwards. The QuestDB server allows 64 connections, so the fixture prunes idle pool connections.

docker compose -f docker-dev/docker-compose.yml -f docker-dev/docker-compose.questdb.yml up -d --no-deps questdb
export HISTORIAN_QUESTDB_TEST_CONNECTION='Host=localhost;Port=18812;Username=admin;Password=quest;Database=qdb;ServerCompatibilityMode=NoTypeLoading'
dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests

---

HistoryUpdate

OPC UA HistoryUpdate lets a client insert, replace, update and delete stored raw history. It is available only when the selected backend provides an updater — SqlServer and QuestDb do; Gateway does not, and an updater is registered only when ServerHistorian:Enabled=true. Updates go straight to the backend (not through the outbox), are Primary-only (#674, below), and share the HistoryRead batch limiter and HistoryReadDeadline.

Operations

OPC UA request	Server override	Per-item results (OperationResults)
UpdateDataDetails, PerformInsertReplace = Insert	HistoryUpdateData	GoodEntryInserted; BadEntryExists if a sample already exists at that timestamp
…Replace	HistoryUpdateData	GoodEntryReplaced; BadNoEntryExists if none exists
…Update	HistoryUpdateData	GoodEntryInserted or GoodEntryReplaced (upsert)
DeleteRawModifiedDetails (IsDeleteModified=false)	HistoryDeleteRawModified	one node-level code: Good; BadNoData if [start, end) held nothing
DeleteAtTimeDetails	HistoryDeleteAtTime	per time: Good; BadNoEntryExists

Each backend call is applied as a unit (SqlServer: one transaction, MERGE … OUTPUT $action against the TVP; QuestDb: an INSERT per chunk plus the WAL-visibility wait, no transaction — see QuestDB backend). Every applied item is also recorded, with the caller's principal, in the modification trail (#668). The node-level StatusCode stays Good when item results are mixed (Part 11 reports item outcomes in OperationResults); it is Bad only for whole-node failures.

Per-item rejections (answered without touching SQL; other items still proceed): a null item ⇒ BadInvalidArgument; SourceTimestamp == DateTime.MinValue ⇒ BadInvalidTimestamp; a SourceTimestamp later than now + ServerHistorian:HistoryUpdateMaxFutureSkew (default 1 day, server clock; #674 — a Local-kind timestamp is converted to UTC first, and is sent as UTC) ⇒ BadOutOfRange — on every backend, never sent; a value the historian cannot type (array, unsupported CLR type) ⇒ BadTypeMismatch; a NaN / ±Infinity Double ⇒ BadInvalidArgument — it cannot live in a SQL float, and unlike the live-value writer (which stores it as Empty) an operator's explicit update is rejected rather than silently rewritten; and a null value with a Good or Uncertain status ⇒ BadInvalidArgument — an Empty sample is always Bad, so a null value is accepted only with a Bad status (an operator-authored outage marker). A caller-supplied status code is otherwise stored as given.

No DataType check. HistoryUpdate does not compare a value's type with the node's DataType: a value is stored as sent (a String written to a Double node's history is stored and later read back as a String), exactly as the live recorder stores whatever type the driver publishes. Only a value the historian cannot represent at all is refused (BadTypeMismatch, above). Part 11 §6.8 permits a server to answer BadTypeMismatch for a value of the wrong type; this server deliberately does not.

DeleteRawModifiedDetails needs both bounds: a StartTime or EndTime of DateTime.MinValue (the wire's "unspecified") is BadInvalidArgument for that node and never reaches the backend.

Duplicate timestamps inside one request (compared at 100 ns): Insert — the first occurrence is applied, later ones are BadEntryExists; Replace / Update — the last one wins and every earlier occurrence reports the winner's code; DeleteAtTime — the first reports the real outcome, later ones BadNoEntryExists. Unknown tag: Insert / Update create the tag row; Replace and the deletes report BadNoEntryExists (DeleteRaw: BadNoData) without creating anything.

Unsupported ⇒ BadHistoryOperationUnsupported (per node): the node is not historized; the backend has no updater (Gateway, or ServerHistorian:Enabled=false); PerformUpdateType.Remove; DeleteRawModifiedDetails.IsDeleteModified=true (the modification trail is an audit record, aged out only by retention); and the out-of-scope detail types — event HistoryUpdate (UpdateEventDetails / DeleteEventDetails) and UpdateStructureData — which are not overridden, so the SDK base rejects them. A backend exception is BadInternalError for that node and a deadline is BadTimeout. On a SqlServer database still at schema v1 the whole node is BadHistoryOperationUnsupported on every arm (#674). No AuditHistoryUpdateEvent is emitted (#674.5, not implemented).

Primary-only (#674)

HistoryUpdate is served only by the node holding the redundancy Primary data-plane role — the same verdict (IDriverPrimaryRoleSource, published by DriverHostActor) that gates native-alarm acknowledges (#597). On any other node every node of the request that passed authorization and is servable is refused BadInvalidState (node-level code, with a message pointing at the Primary) and nothing reaches the backend. Order: authorization first (an unauthorized caller learns BadUserAccessDenied, not the node's role), then the request is planned (a request no server could serve — no updater, a node that is not historized, Remove, IsDeleteModified — is the honest BadHistoryOperationUnsupported on every node, so a Secondary does not send the client to a Primary that would refuse it too), then the gate.

Closed until the first redundancy snapshot (boot window). The gate reads its own verdict, IHistoryUpdateRoleView.ShouldServeHistoryUpdate (PrimaryGatePolicy.ShouldServeHistoryUpdate): the data-plane Primary verdict, but closed until this node has received its first redundancy snapshot. The data-plane verdict itself allows during the pre-join boot window (a node not yet joined sees no driver peer, which reads as "single node") — right for live writes and acks, which must not stall on a genuine single node, but it would let both nodes of a pair accept a HistoryUpdate for a few seconds (up to ~25 s under the bootstrap guard). "No verdict yet" is told apart from "single node" by the snapshot itself: the redundancy singleton runs on every driver node — a lone node included — and publishes on every change and every 10 s heartbeat, so a single node is named Primary within seconds of joining and opens then. Until that first snapshot every node refuses BadInvalidState; after it the gate is exactly the data-plane verdict (including the unknown-role / member-count rule for a snapshot that omits this node). A node with no redundancy manager (admin-only — it registers no IHistoryUpdateRoleView) wires no gate and allows. HistoryRead is not gated. Why: rewriting history is a single-writer operation — both backends capture the prior value for the modification trail under a per-tag, per-process lock (QuestDB) or statement (SqlServer), so two pair nodes updating one tag concurrently could record a wrong prior value. A single-node deployment (no redundancy manager) is unaffected. Clients should send HistoryUpdate to the server advertising the higher ServiceLevel.

The HistoryWrite AccessLevel bit

The SDK base admits a node to HistoryUpdate only if its AccessLevel carries HistoryWrite. The server ORs that bit in only when the node is historized and an updater is wired at materialise time (the raw node and every UNS mirror; the surgical UpdateTagAttributes path recomputes it the same way). OtOpcUaServerHostedService.StartAsync therefore wires the updater before installing the sink, and nodes materialised earlier are never retro-updated. With Backend=Gateway the bit is never set.

Authorization

HistoryUpdate is gated in both ACL modes (rewriting recorded history was never ungated). WriteOperate is deliberately not reused.

Security:Acl:Enabled	Decision
false (default)	The session's API key is re-checked first (AuthorizeKeyUsable, Gitea #563 — an expired or disabled key loses its activation-time roles), then the session must carry the HistoryUpdate role. ApiKeyInterimRoleMapper confers it only from a Cluster-scoped grant carrying NodePermissions.HistoryUpdate; narrower-scoped grants confer nothing (the same under-grant rule as WriteOperate). Denied ⇒ BadUserAccessDenied. A WriteOperate-only key is denied.
true	DataPlaneAuthorizer.Authorize(identity, OpcUaOperation.HistoryUpdate, nodeId, realm) — the scoped NodeAcl grant decides per node and replaces the role check (never chained), so a session with no roles succeeds on the strength of a scoped grant. Denied ⇒ BadUserAccessDenied for that node.

NodePermissions.HistoryUpdate (bit 12) is authorable in the AdminUI ACL editor and is not part of any bundle (ReadOnly / Operator / Engineer / Admin) — it must be granted explicitly. See security.md.

---

Authoring workflow

1. Open the equipment's Tags tab on /uns/equipment/{id}.
2. Create or edit the tag.
   • For typed-editor drivers (Modbus, S7, OpcUaClient, etc.): check the Historize this tag checkbox and, if needed, fill in the Historian tagname (override) textbox.
   • For raw-JSON editors (Galaxy): you can check the same first-class checkboxes (they appear below the JSON textarea), or add "isHistorized":true (and optionally "historianTagname":"...") directly in the textarea.
3. Save and publish. The server rebuilds its address space; the node materialises with Historizing=true and the HistoryRead AccessLevel bit.
4. Confirm with Client.CLI read that the node's Status is Good and that the value is updating. Then issue a historyread to verify the historian connection returns data.

Native-alarm historian opt-out (alarm.historizeToAveva)

A tag carrying a native "alarm" object has a separate historian opt-out for its alarm transitions (distinct from tag-value historization). On the Tag modal, check or uncheck Historize to AVEVA in the alarm section. This maps to alarm.historizeToAveva (bool?) in the TagConfig JSON:

• Absent or true (default) — the alarm's transitions are written to AVEVA Historian via HistorianAdapterActor.
• false — the durable AVEVA write is suppressed for this alarm's transitions. The live /alerts feed and OPC UA condition events are unaffected.

The gate is applied in HistorianAdapterActor using is not false semantics, matching the scripted-alarm HistorizeToAveva posture. See ScriptedAlarms.md §Native driver alarms and AlarmTracking.md §Historian write-back for the full alarm-historian routing.

---

Client.CLI historyread examples

The historyread command reads historical data from any node. Supply start and end times in ISO 8601 UTC form. See docs/Client.CLI.md for the full flag reference.

v3 Batch 4 NodeIds. A historized tag is readable through either of its two NodeIds — the Raw-namespace node (s=<RawPath>, e.g. Plant/Modbus/dev1/Speed) or a referencing equipment's UNS-namespace node (s=<Area>/<Line>/<Equipment>/<EffectiveName>). Both register the same historian tagname, so the returned series is identical. The ns=N index is assigned at connect time per the namespace URI (https://zb.com/otopcua/raw / https://zb.com/otopcua/uns) — resolve it from the server's NamespaceArray rather than hard-coding it. The examples below show a Raw-namespace read (ns=2 illustrative).

# Raw history for a historized tag via its Raw-namespace NodeId (last 24 hours by default)
otopcua-cli historyread \
  -u opc.tcp://localhost:4840/OtOpcUa \
  -n "ns=2;s=Plant/Modbus/dev1/Speed" \
  --start "2026-06-13T00:00:00Z" --end "2026-06-14T00:00:00Z"

# Same series via a referencing equipment's UNS-namespace NodeId, limited to 100 values
otopcua-cli historyread \
  -u opc.tcp://localhost:4840/OtOpcUa \
  -n "ns=3;s=filling/line1/station1/Speed" \
  --start "2026-06-13T00:00:00Z" --end "2026-06-14T00:00:00Z" \
  --max 100

# 1-hour average aggregate
otopcua-cli historyread \
  -u opc.tcp://localhost:4840/OtOpcUa \
  -n "ns=2;s=Plant/Modbus/dev1/Speed" \
  --start "2026-06-13T00:00:00Z" --end "2026-06-14T00:00:00Z" \
  --aggregate Average --interval 3600000

# Authenticated read (ReadOnly role or higher required)
otopcua-cli historyread \
  -u opc.tcp://localhost:4840/OtOpcUa \
  -n "ns=2;s=Plant/Modbus/dev1/Speed" \
  --start "2026-06-13T00:00:00Z" --end "2026-06-14T00:00:00Z" \
  -U reader -P password

Supported --aggregate values: Average, Minimum, Maximum, Count, Start, End, StandardDeviation (aliases: avg, min, max, stddev/stdev, first, last). --interval is the processing interval in milliseconds (default 3600000 = 1 hour). The Total aggregate is served by the server's OPC UA HistoryRead endpoint for any conformant client (derived as time-weighted Average × interval-seconds — see "Total aggregate derivation" above), but is not exposed by this bundled CLI.

---

Live /run gate

The live read gate requires a reachable ZB.MOM.WW.HistorianGateway (VPN to wonder-sql-vd03) with the AVEVA Historian behind it healthy. Set ServerHistorian:Enabled=true with the correct Endpoint (https://host:5222) and supply ServerHistorian__ApiKey via the environment, then deploy and publish at least one historized Galaxy tag. The gate is operator-driven — it is not part of the local docker-dev rig. The gateway-backed driver also ships an env-gated live suite (Category=LiveIntegration); see the HISTGW_GATEWAY_ENDPOINT / HISTGW_GATEWAY_APIKEY / HISTGW_TEST_TAG / HISTGW_WRITE_SANDBOX_TAG / HISTGW_BOOL_SANDBOX_TAG / HISTGW_ALARM_SOURCE env vars (it skips cleanly when they are absent). HISTGW_BOOL_SANDBOX_TAG (e.g. HistGW.LiveTest.BoolSandbox, a dedicated writable Boolean tag — never the shared Float write sandbox) drives the Boolean historization leg (EnsureTags_boolean_sandbox_provisions_as_Int2) and the retype probe (EnsureTags_type_change_outcome_is_observable) that back the Boolean historization maps to Int2 decision. Running the full suite against the gateway (all six tests, none skipped) also discharges 06/U-2's "full documented run" for the historian suite.

See AlarmHistorian.md for the alarm write path and ServiceHosting.md for the (external) HistorianGateway deployment.

---

---

Cross-driver TagConfig key reference

The TagConfig JSON blob is the single extension surface for per-tag server-side behaviours across all drivers. Keys from multiple phases coexist in the same blob; unknown keys are preserved byte-stable on round-trip by all typed editors.

Historian + alarm keys (Phase B / Phase C)

Key	Type	Phase	Description
isHistorized	bool	C	Marks the tag historized (HistoryRead + Historizing=true).
historianTagname	string	C	Explicit historian tagname override (defaults to FullName).
alarm	object	B	Native alarm definition (alarmType, severity, historizeToAveva, …).

Array keys (Phase 4c)

Key	Type	Phase	Description
isArray	bool	4c	When true and arrayLength >= 1, materialises the node as a 1-D array (ValueRank=OneDimension). Absent or false → scalar.
arrayLength	uint (≥1)	4c	Element count; sets ArrayDimensions. Required when isArray: true.

Cross-driver array coverage matrix

Driver	Read mechanism	Live-verify status
Modbus	Contiguous FC03/FC04 block; String/BitInRegister modes supported	Mac-verifiable (sim 10.100.0.35:5020)
S7	ReadBytesAsync block + per-element decode loop	Unit-proven (fixture down)
AB CIP	libplctag native array read (atomic + UDT member arrays)	Unit-proven (fixture down)
AB Legacy	PCCC multi-element file read via libplctag (cap 256 elements)	Unit-proven (fixture down)
TwinCAT	Raw-bytes array read + per-element decode (native notifications carry the whole array)	Live-verified on TwinCAT 2.11 (2026-10-07)

Array writes, multi-dimensional arrays (ValueRank>1), and per-element historization are out of scope — see Uns.md §Array tags.

---

Closed backlog §2 items (Phase 4 / Phase 4b)

The following items from the backlog §2 (archived at docs/plans/2026-06-18-stillpending-backlog-audit.md) were closed by earlier phases and are recorded here so future audits don't re-flag them.

Item	Closed by	Commit
Modbus Int64 / UInt64 OPC UA node DataType correction	Phase 4	bd8fee61
HistoryAggregateType.Total — Total aggregate	Phase 4 (derived client-side as time-weighted Average × interval-seconds)	5e27b5f7
Historian poison alarm-event indefinite retry — dead-letter cap (task #437)	Phase 4	fcb38014

---

See also

• docs/plans/2026-06-14-galaxy-phase-c-historian-design.md — full design and implementation notes
• AlarmHistorian.md — alarm write path; drains to the same HistorianGateway (SendEvent)
• AlarmTracking.md — OPC UA Part 9 alarm surface (event history source)
• Client.CLI.md — full historyread flag reference
• ScriptedAlarms.md §"Native driver alarms" — the Phase B alarm object in TagConfig (parallel carrier)
• Uns.md §Array tags — isArray/arrayLength keys, cross-driver coverage, and deferrals